Project Detail

Automated Data Pipeline Integration and Warehousing Solution

Led the greenfield development of a Python-based ingestion and data integration solution for Azure analytics workflows, turning complex source data into a reliable downstream pipeline.

Role
Data Engineer
Context
Insurance
Domain
Data Engineering
Duration
25 months
Status
Finished

What changed

  1. Designed and built a Python ingestion system that validated, preprocessed, transformed, and routed source files into Azure storage for downstream loading into Synapse.

  2. Implemented an extensible adapter-based architecture that made it easier to support new file formats and handle inconsistent real-world input data without major rework.

  3. Co-architected the underlying Azure infrastructure from scratch and helped align application design with security and provisioning requirements.

  4. Coordinated with data science, infrastructure, and external data providers to align ingestion requirements, clarify source-data expectations, and keep the overall solution operable.

Technology

Python

8 technologies

Azure SDK for Python pandas pyarrow fastparquet PyYAML lxml requests pytest

SQL

1 technology

T-SQL

Terraform

4 technologies

AzureRM Provider AzureAD Provider AzAPI Provider Checkov

Azure

12 technologies

Azure Functions Azure Blob Storage Azure Queue Storage Azure Table Storage Azure Key Vault Azure Synapse Analytics Azure Synapse SQL Pool Azure Virtual Network Azure Private Endpoints Azure Monitor Application Insights Azure Log Analytics Azure Automation

CI/CD Pipelines

2 technologies

GitHub Actions Oryx

Signals

data-engineering cloud-engineering platform-engineering greenfield-development