As a Senior AWS and Databricks/dbt Data Engineer at DataPMI, your role involves designing, building, and maintaining high-performance ETL/ELT pipelines using Databricks (PySpark/SQL) and dbt to transform raw data into curated, analytics-ready models. With 510+ years of experience in Azure cloud ecosystems, you will utilize strong dimensional modelling skills and expertise in implementing data pipelines, data quality, and governance using modern data stack tools. **Key Responsibilities:**
**Data Modelling & Transformation:** Architect and develop advanced data transformation workflows using dbt (models, tests, macros) within the Databricks Medallion Architecture (Bronze/Silver/Gold). - **Pipeline Development:** Build and optimize scalable, high-throughput, and low-latency data pipelines using Azure Databricks (PySpark, SQL), dbt, Airflow, and Azure Data Lake Storage (ADLS). - **Analytics Engineering:** Translate complex business requirements into data models (Star/Snowflake schemas) to support BI reporting, ad-hoc analysis, and AI/ML initiatives. - **Code & Quality Control:** Implement CI/CD for dbt using GitHub/Azure DevOps, ensuring rigorous testing, documentation, and data quality checks (using Unity Catalog). - **Performance Tuning:** Optimize Spark jobs, SQL queries, and dbt models for maximum efficiency and cost-performance. - **Mentorship:** Provide technical guidance and conduct code reviews for junior engineers. **Required Technical Skills:**
**Cloud & ETL:** Extensive experience with Azure Databricks or similar cloud environments in medallion architecture
**Languages:** Strong Python and SQL skills
**Engineering Practices:** Git/GitHub, CI/CD, and data modelling (Kimball methodology)
**Data Governance:** Experience with Unity Catalogue or similar data cataloguing/governance tools
**Good to Have:**
Experience with the creation of packages and working with SDK
Background in domain-specific analytics (e.g., Retail, Manufacturing)
Certification in Databricks
**Key Competencies:**
**Collaboration:** Working with Data Scientists, Data Analysts, and business stakeholders
**Problem-Solving:** Ability to debug complex data pipeline failures and optimize performance
**Communication:** Strong verbal and written communication skills to articulate technical designs
If you are interested in this exciting opportunity, apply using the Job Code: EZW-100123 at Eazyway Job Portal. Hurry up and sign up to apply for this position! As a Senior AWS and Databricks/dbt Data Engineer at DataPMI, your role involves designing, building, and maintaining high-performance ETL/ELT pipelines using Databricks (PySpark/SQL) and dbt to transform raw data into curated, analytics-ready models. With 510+ years of experience in Azure cloud ecosystems, you will utilize strong dimensional modelling skills and expertise in implementing data pipelines, data quality, and governance using modern data stack tools. **Key Responsibilities:**
**Data Modelling & Transformation:** Architect and develop advanced data transformation workflows using dbt (models, tests, macros) within the Databricks Medallion Architecture (Bronze/Silver/Gold). - **Pipeline Development:** Build and optimize scalable, high-throughput, and low-latency data pipelines using Azure Databricks (PySpark, SQL), dbt, Airflow, and Azure Data Lake Storage (ADLS). - **Analytics Engineering:** Translate complex business requirements into data models (Star/Snowflake schemas) to support BI reporting, ad-hoc analysis, and AI/ML initiatives. - **Code & Quality Control:** Implement CI/CD for dbt using GitHub/Azure DevOps, ensuring rigorous testing, documentation, and data quality checks (using Unity Catalog). - **Performance Tuning:** Optimize Spark jobs, SQL queries, and dbt models for maximum efficiency and cost-performance. - **Mentorship:** Provide technical guidance and conduct code reviews for junior engineers. **Required Technical Skills:**