Posted May 14, 2026
We are seeking an experienced Data Engineering Manager to lead the technical evolution of our Databricks Lakehouse. You will be responsible for the architecture, scaling, and reliability of our data products, ensuring that the platform provides high-quality, actionable data for retail and supply chain analytics. This is a leadership role that requires a balance of advanced systems design and hands-on operational rigor. ### Key Responsibilities
Lakehouse Strategy: Own the end-to-end design and optimization of the Databricks Lakehousearchitecture, leveraging Unity Catalog and Delta Lake for multi-tenant isolation. - Scalable Ingestion: Drive multi-tenant ingestion strategies from Azure Table Storage, Cosmos DB, and APIs, focusing on watermark logic and idempotent pipelines. - Framework Development: Lead the evolution of in-house PySpark-based ETL frameworks to support complex transformations and downstream consumption. #### ๐น Operational Excellence & Data Observability
Data Quality Validation: Implement comprehensive data validation frameworks and automated checks to ensure accuracy, completeness, and consistency across the Lakehouse. - Observability & Monitoring: Drive the strategy for end-to-end data observability, using monitoring and alerting to proactively identify pipeline drifts or failures before they impact stakeholders. - Improve cost transparency through DBU tracking and compute profiling (Serverless vs. Classic); drive performance tuning via clustering and partitioning strategies. #### ๐น DevOps & Technical Governance
Deployment Integrity: Oversee CI/CD pipelines and Databricks Asset Bundles (DABs) to maintain parity between UAT and Production environments. - Governance Standards: Implement version control, rollback strategies, and deployment governance to ensure a stable and auditable production environment. #### ๐น Cross-Functional Leadership
Strategic Partner: Partner with BI, Data Science, and Product teams to bridge business requirements into scalable, production-grade data solutions. - Team Growth: Mentor and lead a distributed team of engineers, focusing on high-velocity delivery and technical excellence. - Tenant Logic: Support customer onboarding by designing tenant-specific transformation logic that scales without increasing architectural complexity. ### Qualifications
8+ years in Data Engineering; 3+ years in a leadership/management role. - Expert in Workflows, Unity Catalog, and Delta Lake. - Deep proficiency in PySpark, SQL, and Azure Data ecosystem (Storage, Cosmos DB). - Proven experience managing multi-tenant or enterprise-scale platforms. - Strong background in CI/CD, Infrastructure-as-Code, and Data Modeling. ### Preferred Skills
Experience with retail or supply chain data forecasting. - Expertise in optimizing serverless compute environments. - Experience maintaining production SLAs for customer-facing data dependencies.
Don't want to apply yourself?
Our team writes your resume, applies for you, preps you for interviews, and negotiates your offer.
Browse Jobs
By Role
By City