Lead Data Engineer

Familiarity with Agentic AI concepts for automation, data quality validation, and metadata enrichment will be advantageous. Required Skills & Experience:

6+ years of experience in data engineering, with a minimum of 1 year in Databricks. - Hands-on expertise in:

Databricks (PySpark, Spark SQL, Delta Lake, Unity Catalog, Delta Live tables). - SQL for data modeling and query optimization. - AWS services (S3, Glue, Kinesis, Redshift, API Gateway, SNS, Lambda). - Airflow for workflow orchestration. - Added advantage DBT, Splunk, Atlan, Java, API, Power BI. - Strong analytical and problem-solving skills with meticulous attention to detail. Preferred Qualifications:

Knowledge of Agentic AI concepts and frameworks. Join our team and take on this challenging role where you can contribute to building cutting-edge data solutions and drive impactful business decisions. As a Lead Data Engineer at our company located in Bengaluru, your primary responsibility will be designing, building, and optimizing data pipelines and platforms to facilitate advanced analytics and business intelligence. You will need to demonstrate expertise in Databricks, SQL, AWS, and Airflow, in addition to possessing a strong foundation in data analysis and engineering best practices. Your collaboration with cross-functional stakeholders will be essential to ensure the development of scalable, secure, and high-performing data solutions. Key Responsibilities:

Technical Design:

Develop and maintain data pipelines and ETL workflows using Databricks and Airflow. - Possess a good understanding of Databricks Delta Live tables and Unity Catalog. - Build frameworks and templates for ETL/ELT, specifically for telemetry pipelines. - Ensure that cloud-native designs adhere to AWS platform best practices. - Governance, Security & Optimization:

Ensure data quality, integrity, and governance across all data platforms. - Optimize SQL queries and data models for enhanced performance and scalability. - Implement monitoring and observability for data workflows. - Cross-Functional Collaboration:

Collaborate closely with architects and leads to comprehend data requirements and develop technical solutions. - Work with analytics and business teams to deliver actionable insights through BI dashboards and reports. - AI/LLM Enablement (Advantage):

Familiarity with Agentic AI concepts for automation, data quality validation, and metadata enrichment will be advantageous. Required Skills & Experience:

6+ years of experience in data engineering, with a minimum of 1 year in Databricks. - Hands-on expertise in:

Databricks (PySpark, Spark SQL, Delta Lake, Unity Catalog, Delta Live tables). - SQL for data modeling and query optimization. - AWS services (S3, Glue, Kinesis, Redshift, API Gateway, SNS, Lambda). - Airflow for workflow orchestration. - Added advantage DBT, Splunk, Atlan, Java, API, Power BI. - Strong analytical and problem-solving skills with meticulous attention to detail. Preferred Qualifications:

Knowledge of Agentic AI concepts and frameworks. Join our team and take on this challenging role where you can contribute to building cutting-edge data solutions and drive impactful business decisions.