Data Engineering Operations Lead

TransUnion's Job Applicant Privacy Notice What We'll Bring: What We'll Bring: About TransUnion: TransUnion is a global information and insights company which provides solutions that help create economic opportunity, great experiences, and personal empowerment for hundreds of millions of people in more than 30 countries. We call this Information for Good®. TransUnion is a leading credit reference agency, and we offer specialist services in fraud, identity, and risk management, automated decisioning and demographics. We support organizations across a wide variety of sectors including finance, retail, telecommunications, utilities, gaming, government, and insurance. What You'll Bring: We are looking for a Lead Data Engineer to join our growing Data Engineering and Analytics Practice who will drive building next generation suite of products and platform by designing, coding, building, and deploying highly scalable and robust solutions. You will be based both from our offices in Pune and working remotely as part of our 'flex together' approach. In this fast-paced role you will work with Business Stakeholders to achieve business goals. This exciting role will offer a host of development opportunities as part of a growing global business. What You'll Bring: Key Responsibilities: Own and manage daily Data Engineering operations, including data ingestion workflows, pipeline monitoring, and incident resolution. Oversee end-to-end pipeline health - proactively monitor, triage, and resolve failures, bottlenecks, and data quality issues across all ingestion layers. Organize and prioritize the team's daily workload - assign tasks, track progress, and remove blockers to ensure smooth operational delivery. Maintain and improve ingestion pipelines from various data sources into data lakes, warehouses, and real-time streaming systems. Act as the first point of escalation for pipeline failures, SLA breaches, and data anomalies - driving root cause analysis and permanent fixes. Define and enforce operational standards - runbooks, alerting thresholds, on-call procedures, and incident management practices. Coordinate with upstream data providers and downstream consumers to manage dependencies and communicate pipeline status. Lead daily stand-ups, sprint planning, and work organization for the Data Engineering Ops function under the wider Data Engineering practice Mentor and guide junior engineers on operational best practices, debugging, and pipeline development. Collaborate with data scientists, analysts, and product partners to onboard new data sources and meet ingestion SLAs. Drive continuous improvement in pipeline reliability, observability, and efficiency. Implement best practices in data governance, data quality monitoring, and compliance across all pipelines. Design, build, test, and deploy Data solutions at scale, including data lakes, data warehouses, and real-time analytics. Lead technical delivery on use cases, plan and delegate tasks to junior team members, and oversee work from inception to final product. Skills & Experience: Essential: Bachelor's degree in Computer Science, Engineering, Statistics or a related field Minimum 8 years of data engineering experience (ideally 10+), with at least 3 years in senior/lead roles. 6+ years of experience in Big Data technologies (e.g., Spark, Hive, Hadoop, Databricks). Advanced proficiency with Apache Spark, including PySpark and SparkSQL, tuning and performance optimisation experience is fundamental. Proficiency in Python, Pandas (Scala/Java knowledge is desirable). Working knowledge of Apache Hive. Strong SQL knowledge and experience (T-SQL, working with SQL Server, SSMS). Expertise in designing and implementing scalable data pipelines and ETL processes using the GCP data stack, including BigQuery, Dataflow, Pub/Sub, Cloud Storage, Cloud Composer, Cloud Functions, Dataproc (Spark). Excellent knowledge of data engineering concepts and best practices. Proven ability to lead, mentor, inspire, and support junior team members. Ability to lead technical deliverables autonomously and guide junior data engineers. Ability to organize and manage daily engineering workload - task assignment, prioritization, and delivery tracking. Strong attention to detail and adherence to best practices. Experience with batch, real-time streaming, and ETL processes, including incident resolution and pipeline recovery. Experience building and managing ETL workflows using Apache Airflow, including DAG creation, scheduling, and error handling. Source control with Git. Knowledge of CI/CD concepts and experience designing CI/CD for data pipelines. Experience designing logical data models and physical data models, including data warehouse and data mart designs. Knowledge of Delta Lake concepts and common data formats, Lakehouse architecture. Software engineering principles including OOP, design patterns, SDLC, Agile, TDD, and performance optimization. Desirable: Relevant certifications (e.g., Google Cloud Professional Data Engineer). Experience with streaming services such as Kafka is a plus. R & Sparklyr experience is a plus. Knowledge of MLOps concepts, AI/ML lifecycle management, and MLflow. Harness experience is a plus. Impact You'll Make: Impact You'll Make: TransUnion – a place to grow: We know that it is unrealistic to expect candidates to have each and every aspect of the essential and/or desirable skills listed above – if there is something you can't tick off right now – good, you can learn here! Impact you will make: Enable Decision Making across the organization using data driven culture. What's In It for you? At TransUnion you will be joining a friendly, forward thinking global business. As well as a competitive salary & bonus scheme our benefits package includes up 26 days' annual leave (plus bank holidays) a generous contributory pension scheme, private health care and a host of other employee lifestyle benefits. That is in addition to a variety of physical, mental, and financial fitness wellbeing programmes such as lunchtime yoga, boxing classes, mindfulness app access, daily dedicated 'away from keyboard' time to ensure colleagues take a break and our diversity forums and networking groups. This is a hybrid position and involves regular performance of job responsibilities virtually as well as in-person at an assigned TU office location for a minimum of two days a week. We are committed to being a place where diversity is not only present, but also embraced. This is a hybrid position and involves regular performance of job responsibilities virtually as well as in-person at an assigned TU office location for a minimum of two days a week. TransUnion Job Title Lead Engineer, Data Analysis Beware of scams tied to potential career opportunities. If you received a link, email, text or social media communication you weren't expecting from TransUnion, take a moment to confirm the relevant information. Be wary when providing personal information. Contact careers@transunion.com or (312) 258-1717 for assistance. View TransUnion's Job Applicant Privacy Notice TransUnion is a global information and insights company with over 12,000 associates operating in more than 30 countries. We make trust possible by ensuring each person is reliably represented in the marketplace. We do this with a Tru™ picture of each person: an actionable view of consumers, stewarded with care. Through our acquisitions and technology investments we have developed innovative solutions that extend beyond our strong foundation in core credit into areas such as marketing, fraud, risk and advanced analytics. As a result, consumers and businesses can transact with confidence and achieve great things. We call this Information for Good® — and it leads to economic opportunity, great experiences and personal empowerment for millions of people around the world.