As a Data Engineer III at Withum, you will play a crucial role in designing, building, and optimizing data pipelines and architectures while mentoring junior engineers and collaborating across teams to deliver scalable data solutions. You will spend your time on the following key responsibilities:
Designing and implementing scalable data pipelines using Azure Data Factory, Synapse Pipelines, and Apache Spark on Azure Fabric. - Architecting and maintaining data models and storage solutions using OneLake, Delta Lake, Parquet, and ADLS gen2. - Developing and optimizing Dataflows Gen2, Notebooks, and KQL queries for data transformation and exploration. - Collaborating with data scientists, analysts, and business stakeholders to understand data needs and deliver robust solutions. - Implementing data governance, lineage, and quality frameworks using Microsoft Purview and Fabric Data Activator. - Supporting CI/CD pipelines and automation for data workflows using Azure DevOps or GitHub Actions. - Integrating external datasets by working with APIs and third-party data sources. - Ensuring security, compliance, and performance of data systems. - Mentoring junior engineers and contributing to best practices and architectural standards. The ideal candidate for this role should possess the following qualifications:
Education: A bachelor's degree or higher in Computer Science, Information Systems, or a related field. - Experience: 8 years of experience in data engineering, including a minimum of two years on Azure. - Proficiency: Skilled in SQL, Python, and Spark. - Azure Services: Experience with Azure cloud services such as Data Factory, Synapse, Blob Storage, Functions, ADLS gen2, and Data Bricks. - Microsoft Fabric: Advanced expertise in Microsoft Fabric. - Data Management: Familiarity with data governance, lineage, and cataloging tools. - Data Warehousing: Strong understanding of data warehousing concepts and dimensional modeling. - Methodologies: Proficient with version control (Git), CI/CD, and Agile methodologies. - Communication: Excellent communication and collaboration skills. - Certification: Microsoft Azure Data Engineer certification is an advantage. Additionally, the must-have technical skills required for the role include:
Cloud Platform: Advanced expertise in Microsoft Azure, with a particular focus on Microsoft Fabric. - Data Engineering: Proficiency in Azure Data Factory, Azure Synapse Analytics, and Apache Spark. - Lakehouse Architecture: Familiarity with OneLake, Delta Lake, Parquet, Delta, Iceberg, and Hudi. - Warehousing: Proficiency in star/snowflake schemas, dimensional modeling, and satellite hub models. - Programming: Expertise in Python, PySpark, SQL, and KQL (Kusto Query Language). - Data Orchestration: Firsthand experience with Dataflows Gen2 and Notebooks. - DevOps & CI/CD: Proficiency in Azure DevOps, Git, and YAML pipelines. - Security & Governance: Advanced understanding of Microsoft Purview, RBAC, and data masking techniques. Nice-to-have skills for the role include knowledge and experience with Azure Event Hubs, Azure Stream Analytics, Azure ML, MLflow, Azure Monitor, Log Analytics, Power BI, Bicep, Terraform, Data Activator, dbt, Delta Live Tables, Great Expectations, among others. In conclusion, at Withum, you will be part of a team where talent thrives, and you will have the opportunity to work with innovative tools and solutions to address data, analytics, and business intelligence needs. For more information about Withum, you can visit their website at www.withum.com. As a Data Engineer III at Withum, you will play a crucial role in designing, building, and optimizing data pipelines and architectures while mentoring junior engineers and collaborating across teams to deliver scalable data solutions. You will spend your time on the following key responsibilities:
Designing and implementing scalable data pipelines using Azure Data Factory, Synapse Pipelines, and Apache Spark on Azure Fabric. - Architecting and maintaining data models and storage solutions using OneLake, Delta Lake, Parquet, and ADLS gen2. - Developing and optimizing Dataflows Gen2, Notebooks, and KQL queries for data transformation and exploration. - Collaborating with data scientists, analysts, and business stakeholders to understand data needs and deliver robust solutions. - Implementing data governance, lineage, and quality frameworks using Microsoft Purview and Fabric Data Activator. - Supporting CI/CD pipelines and automation for data workflows using Azure DevOps or GitHub Actions. - Integrating external datasets by working with APIs and third-party data sources. - Ensuring security, compliance, and performance of data systems. - Mentoring junior engineers and contributing to best practices and architectural standards. The ideal candidate for this role should possess the following qualifications:
Education: A bachelor's degree or higher in Computer Science, Information Systems, or a related fi
