You have a job opportunity for a Databricks Engineer with 6 to 8 years of relevant experience to join the data engineering team and contribute to designing and implementing scalable data solutions on the Azure platform. As a Databricks Engineer, you will work closely with cross-functional teams to build high-performance data pipelines and maintain a modern Lakehouse architecture. Key Responsibilities:
Design and develop scalable data pipelines using Spark-SQL and PySpark in Azure Databricks. - Build and maintain Lakehouse architecture using Azure Data Lake Storage (ADLS) and Databricks. - Perform comprehensive data preparation tasks, including:
Data cleaning and normalization
Deduplication
Type conversions
Collaborate with the DevOps team to deploy and manage solutions in production environments. - Partner with Data Science and Business Intelligence teams to share insights, align on best practices, and drive innovation. - Support change management through training, communication, and documentation during upgrades, data migrations, and system changes. Required Qualifications:
5+ years of IT experience with strong exposure to cloud technologies, particularly in Microsoft Azure. - Hands-on experience with:
Databricks, Azure Data Factory (ADF), and Azure Data Lake Storage (ADLS)
Programming with PySpark, Python, and SQL
Solid understanding of data engineering concepts, data modeling, and data processing frameworks. - Ability to work effectively in distributed, remote teams. - Excellent communication skills in English (both written and verbal). Preferred Skills:
Strong working knowledge of distributed computing frameworks, especially Apache Spark and Databricks. - Experience with Delta Lake and Lakehouse architecture principles. - Familiarity with data tools and libraries such as Pandas, Spark-SQL, and PySpark. - Exposure to on-premise databases such as SQL Server, Oracle, etc. - Experience with version control tools (e.g., Git) and DevOps practices including CI/CD pipelines. Key Responsibilities:
Design and develop scalable data pipelines using Spark-SQL and PySpark in Azure Databricks. - Build and maintain Lakehouse architecture using Azure Data Lake Storage (ADLS) and Databricks. - Perform comprehensive data preparation tasks, including:
Data cleaning and normalization
Deduplication
Type conversions
Collaborate with the DevOps team to deploy and manage solutions in production environments. - Partner with Data Science and Business Intelligence teams to share insights, align on best practices, and drive innovation. - Support change management through training, communication, and documentation during upgrades, data migrations, and system changes. Required Qualifications:
5+ years of IT experience with strong exposure to cloud technologies, particularly in Microsoft Azure. - Hands-on experience with:
Databricks, Azure Data Factory (ADF), and Azure Data Lake Storage (ADLS)
Programming with PySpark, Python, and SQL
Solid understanding of data engineering concepts, data modeling, and data processing frameworks. - Ability to work effectively in distributed, remote teams. - Excellent communication skills in English (both written and verbal). Preferred Skills:
Strong working knowledge of distributed computing frameworks, especially Apache Spark and Databricks. - Experience with Delta Lake and Lakehouse architecture principles. - Familiarity with data tools and libraries such as Pandas, Spark-SQL, and PySpark. - Exposure to on-premise databases such as SQL Server, Oracle, etc. - Experience with version control tools (e.g., Git) and DevOps practices including CI/CD pipelines.
