Professional Insurance Featured

Automated Data Pipeline Integration and Warehousing Solution

Cloud Developer 2 years 1 month

Led the development of an automated data pipeline to support customer's Azure Data Warehousing needs, reducing manual ETL efforts by 80%.

Key Responsibilities

  • Developed a robust Python API to manage data workflows, including resolving issues in received files, preprocessing data, and converting it into Parquet format for efficient loading into Azure Synapse.
  • Set up the infrastructure for Azure Synapse and Functions, adhering to best practices for scalable data warehousing.
  • Automated ETL pipelines that processed tens of thousands of records daily.
  • Ensured infrastructure updates and data pipeline maintenance aligned with customer priorities.

Technology Stack

Python
azure-sdk pyarrow fastparquet pandas pytest
SQL Terraform
Azure
Azure Function Apps Azure Synapse Azure Networking Azure Storage
CI/CD Pipelines
GitHub Actions

Tags

#professional #cloud #data-engineering