Build and maintain an end-to-end Lakehouse using the Medallion architecture (Bronze, Silver, Gold layers). Implement open table formats like Apache Iceberg or Delta Lake to provide ACID transactions, time travel, and schema evolution. - DISTRIBUTED QUERY EXCELLENCE
Design and optimize Starburst/Trino (formerly PrestoSQL) clusters to enable high-performance federated queries across multiple data sources, minimizing data movement and latency. - MULTI-TENANCY & ISOLATION
Engineer robust data pipelines that support multi-tenant environments, ensuring strict data isolation, security, and resource management across different client organizations. - PIPELINE ENGINEERING
Develop scalable batch and streaming pipelines using Apache Spark (PySpark/Scala) and Kafka. Automate orchestration using tools like Airflow or Dagster. - PERFORMANCE TUNING
Conduct deep-dive root cause analysis into pipeline bottlenecks. Optimize Spark jobs, Trino queries, and storage layouts (partitioning, compaction, and vacuuming) to reduce compute costs and improve speed. - DATA RELIABILITY & GOVERNANCE
Implement "privacy-by-design" principles, including encryption, IAM least-privilege, and data masking. Define and monitor SLOs for data freshness, quality, and completeness. Qualifications:
Education: Bachelors or Masters degree in Computer Science, Software Engineering, or a related technical field. - Experience: 10+ years of experience in Data Engineering, with a proven track record of building and scaling production-grade data platforms. - Leadership: Experience mentoring junior engineers through code reviews, design docs, and
(Note: Additional details of the company were not present in the provided job description) Role Overview:
As a senior Data Engineer, you will design and implement high-scale, multi-tenant Data Lakehouse for our clients that serves as the backbone for both real-time analytics and advanced machine learning. Your mission is to move beyond traditional ETL, creating a "data supply chain" that leverages federated query engines, open table formats, and cloud-native scalability to ensure our customers' data is reliable, governed, and instantly accessible across diverse product lines. Key Responsibilities:
LAKEHOUSE ARCHITECTURE
Build and maintain an end-to-end Lakehouse using the Medallion architecture (Bronze, Silver, Gold layers). Implement open table formats like Apache Iceberg or Delta Lake to provide ACID transactions, time travel, and schema evolution. - DISTRIBUTED QUERY EXCELLENCE
Design and optimize Starburst/Trino (formerly PrestoSQL) clusters to enable high-performance federated queries across multiple data sources, minimizing data movement and latency. - MULTI-TENANCY & ISOLATION
Engineer robust data pipelines that support multi-tenant environments, ensuring strict data isolation, security, and resource management across different client organizations. - PIPELINE ENGINEERING
Develop scalable batch and streaming pipelines using Apache Spark (PySpark/Scala) and Kafka. Automate orchestration using tools like Airflow or Dagster. - PERFORMANCE TUNING
Conduct deep-dive root cause analysis into pipeline bottlenecks. Optimize Spark jobs, Trino queries, and storage layouts (partitioning, compaction, and vacuuming) to reduce compute costs and improve speed. - DATA RELIABILITY & GOVERNANCE
Implement "privacy-by-design" principles, including encryption, IAM least-privilege, and data masking. Define and monitor SLOs for data freshness, quality, and completeness. Qualifications:
Education: Bachelors or Masters degree in Computer Science, Software Engineering, or a related technical field. - Experience: 10+ years of experience in Data Engineering, with a proven track record of building and scaling production-grade data platforms. - Leadership: Experience mentoring junior engineers through code reviews, design docs, and
(Note: Additional details of the company were not present in the provided job description)
