As a Senior Big Data Engineer at our company, you will have the opportunity to join dunnhumby's Data Engineering Team, a market-leading business that values innovation and impact. Your role will involve designing end-to-end data solutions, architecting scalable data pipelines, and ensuring data integrity and availability. Here is a breakdown of your key responsibilities:
Design end-to-end data solutions, encompassing data lakes, data warehouses, ETL/ELT pipelines, APIs, and analytics platforms. - Architect scalable and low-latency data pipelines using tools like Apache Kafka, Flink, or Spark Streaming for handling high-velocity data streams. - Design and orchestrate end-to-end automation with frameworks like Apache Airflow to manage complex workflows and dependencies. - Create intelligent systems capable of detecting anomalies, triggering alerts, and autonomously maintaining data integrity. - Implement data governance, metadata management, and data quality standards. - Lead architectural reviews and technical design sessions to drive solution development. - Collaborate with business and IT teams to translate business needs into data architecture requirements. - Ensure security, compliance, and regulatory requirements are met in all data solutions. - Evaluate and propose enhancements to existing data architecture and processes. In addition to these responsibilities, you are expected to have the following technical expertise:
Bachelor's or master's degree in computer science, Information Systems, Data Science, or a related field. - Extensive experience with high-level programming languages such as Python, Java, or Scala. - 5+ years of experience in data architecture, data engineering, or related fields. - Proficiency in data pipeline tools like Apache Spark, Kafka, Airflow, or similar. - Familiarity with data governance frameworks and tools (e.g., Collibra, Alation, OpenMetadata). - Strong knowledge of cloud platforms (Azure or Google Cloud) and cloud-native data services. - Experience in Agile or DevOps environments. - Familiarity with modern data stack tools such as dbt, Snowflake, Databricks. - Hands-on experience with Hive, Oozie, Airflow, HBase, MapReduce, Spark, and working knowledge of Hadoop/Spark Toolsets. - Extensive experience working with Git and Process Automation. - In-depth understanding of relational database management systems (RDBMS) and Data Flow Development. Join us in this exciting opportunity to expand your career and make a significant impact in the field of Big Data Engineering. As a Senior Big Data Engineer at our company, you will have the opportunity to join dunnhumby's Data Engineering Team, a market-leading business that values innovation and impact. Your role will involve designing end-to-end data solutions, architecting scalable data pipelines, and ensuring data integrity and availability. Here is a breakdown of your key responsibilities:
Design end-to-end data solutions, encompassing data lakes, data warehouses, ETL/ELT pipelines, APIs, and analytics platforms. - Architect scalable and low-latency data pipelines using tools like Apache Kafka, Flink, or Spark Streaming for handling high-velocity data streams. - Design and orchestrate end-to-end automation with frameworks like Apache Airflow to manage complex workflows and dependencies. - Create intelligent systems capable of detecting anomalies, triggering alerts, and autonomously maintaining data integrity. - Implement data governance, metadata management, and data quality standards. - Lead architectural reviews and technical design sessions to drive solution development. - Collaborate with business and IT teams to translate business needs into data architecture requirements. - Ensure security, compliance, and regulatory requirements are met in all data solutions. - Evaluate and propose enhancements to existing data architecture and processes. In addition to these responsibilities, you are expected to have the following technical expertise:
Bachelor's or master's degree in computer science, Information Systems, Data Science, or a related field. - Extensive experience with high-level programming languages such as Python, Java, or Scala. - 5+ years of experience in data architecture, data engineering, or related fields. - Proficiency in data pipeline tools like Apache Spark, Kafka, Airflow, or similar. - Familiarity with data governance frameworks and tools (e.g., Collibra, Alation, OpenMetadata). - Strong knowledge of cloud platforms (Azure or Google Cloud) and cloud-native data services. - Experience in Agile or DevOps environments. - Familiarity with modern data stack tools such as dbt, Snowflake, Databricks. - Hands-on experience with Hive, Oozie, Airflow, HBase, MapReduce, Spark, and working knowledge of Hadoop/Spark Toolsets. - Extensive experience working with Git and Process Automation. - In-depth understanding of relational database management systems (RDBMS) and Data Flow Development.