As a Data Engineer - ETL at Algoworks, your primary responsibility will be to design, build, and optimize scalable data pipelines using Azure cloud technologies. You will focus on developing robust data ingestion and transformation pipelines, implementing Delta Lake-based data architectures, and enabling high-quality curated datasets for downstream analytics and reporting. Your expertise in PySpark, Azure Databricks, and Azure Data Factory, along with a deep understanding of data performance optimization and engineering best practices, will be crucial for success in this role. **Key Responsibilities:**
**Pipeline Development**
Build and maintain scalable data pipelines using Azure Databricks and Azure Data Factory. - Implement ingestion and transformation logic across Bronze (raw) and Silver (cleaned) layers. - Support batch and incremental data processing patterns. - **Curated Layer & Data Processing**
Implement hydration, merge, and upsert logic using Delta Lake. - Ensure curated datasets meet business requirements and data quality standards. - Handle late-arriving data and incremental updates efficiently. - **Performance & Storage Optimization**
Optimize Delta Lake tables for performance and cost efficiency. - Select and tune appropriate storage formats (Parquet, Delta). - Apply partitioning, compaction, and file sizing strategies. - Tune Spark jobs for large-scale distributed data processing. - **Downstream Collaboration & Data Enablement**
Collaborate with DWH and BI teams to support downstream data consumption. - Provide optimized datasets for Synapse and reporting workloads. - Support data validation, reconciliation, and consistency across Gold layer outputs. - **Engineering Best Practices**
Implement CI/CD practices for data pipelines and workflows. - Follow coding standards, documentation, and version control practices. - Support production troubleshooting, monitoring, and performance tuning. **Required Skills & Qualifications:**
Bachelors degree in computer science, Engineering, or related field. - 68 years of experience in data engineering. - Strong expertise in:
PySpark and distributed data processing. - Azure Databricks (hands-on development and optimization). - Azure Data Factory for pipeline orchestration. - Deep knowledge of Delta Lake (merge, upsert, optimization techniques). - Strong SQL skills for data transformation and validation. - Experience handling large datasets in distributed environments. - Strong understanding of storage optimization (Parquet, Delta). - Tools & Practices:
Experience with Git and version control systems. - Familiarity with CI/CD pipelines for data workflows. - Understanding of data quality checks and validation techniques. - Experience working in Agile/Scrum delivery models. **Nice to Have Skills:**
Experience supporting Synapse Dedicated SQL Pool. - Exposure to streaming or near real-time data pipelines. - Familiarity with data governance or metadata management tools. **Soft Skills & Collaboration:**
Strong analytical and problem-solving skills. - Ability to work independently on complex data pipelines. - Good communication and collaboration skills. - Proactive and ownership-driven mindset. **Desired Attributes:**
Strong attention to data quality and performance. - Continuous learning mindset for evolving cloud/data technologies. - Ability to work in fast-paced, data-intensive environments. As a Data Engineer - ETL at Algoworks, your primary responsibility will be to design, build, and optimize scalable data pipelines using Azure cloud technologies. You will focus on developing robust data ingestion and transformation pipelines, implementing Delta Lake-based data architectures, and enabling high-quality curated datasets for downstream analytics and reporting. Your expertise in PySpark, Azure Databricks, and Azure Data Factory, along with a deep understanding of data performance optimization and engineering best practices, will be crucial for success in this role. **Key Responsibilities:**
**Pipeline Development**
Build and maintain scalable data pipelines using Azure Databricks and Azure Data Factory. - Implement ingestion and transformation logic across Bronze (raw) and Silver (cleaned) layers. - Support batch and incremental data processing patterns. - **Curated Layer & Data Processing**
Implement hydration, merge, and upsert logic using Delta Lake. - Ensure curated datasets meet business requirements and data quality standards. - Handle late-arriving data and incremental updates efficiently. - **Performance & Storage Optimization**
Optimize Delta Lake tables for performance and cost efficiency. - Select and tune appropriate storage formats (Parquet, Delta). - Apply partitioning, compaction, and file sizing strategies. - Tune Spark jobs for large-scale distributed data processing. - **Downstream Collaboration & Data Enablement**
