As a Lead Data Ingestion Engineer at Innocap, you will play a crucial role in designing and leading scalable, secure, and automated data ingestion frameworks. With your 710+ years of experience, you will be responsible for mentoring a team of data engineers, defining architectural best practices, ensuring the reliability of ingestion ecosystems, and collaborating closely with data platform, analytics, and business teams. **Key Responsibilities:**
Lead the design and development of enterprise-grade ingestion architectures using Databricks, SSIS, Azure Functions, and cloud services. - Drive modernization initiatives such as SSIS-to-ADF migration, batch-to-real-time ingestion, and automation improvements. - Ensure cost optimization, observability, data quality, and governance best practices. - Build and enhance ETL/ELT workflows using Python, Databricks (PySpark), SQL, and Azure components. - Develop C#/.NET services or functions for ingestion, validation, and integration with APIs/microservices. - Optimize ingestion workflows for performance, cost, and scalability. - Mentor and guide junior and mid-level data engineers. - Conduct architectural reviews, code reviews, and provide technical coaching. - Partner with Architects, DevOps Engineers, QA, and business stakeholders to align ingestion solutions with enterprise strategy. **Qualifications Required:**
Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field. - 710+ years of experience in data engineering/data ingestion, with at least 23 years in a lead role. - Deep experience with Databricks, Azure Functions, SSIS, and Python. - Strong programming background in Python and C#/.NET. - Solid hands-on experience with SQL Server, T-SQL, and performance tuning. - Experience building CI/CD pipelines with Azure DevOps. - Strong Python development skills for ingestion, automation, and transformation. - Experience migrating SSIS workloads to Azure-native tools. - Hands-on experience with Databricks (PySpark) for distributed data processing. - Solid C#, .NET background for integration and ingestion services. - Strong SQL and relational database ingestion experience. - Experience with Git, CI/CD, and DevOps automation. - Experience with Delta Lake, Lakehouse architecture, or Synapse. - Experience with Docker/Kubernetes for containerized workloads. - Knowledge of data governance, lineage, and security practices. - Domain knowledge in BFSI or financial datasets. **Additional Company Details:**
Innocap is a leading firm of managed account platform services with over US$100 billion in assets under management, shaping the future of alternative investments for institutional owners and allocators. The company prides itself on the diversity of its people and the strength of their ambitions, offering a hybrid work culture, a smart, talented & agile team, competitive compensation package, continuous learning opportunities, and a commitment to diversity and inclusion. As a Lead Data Ingestion Engineer at Innocap, you will play a crucial role in designing and leading scalable, secure, and automated data ingestion frameworks. With your 710+ years of experience, you will be responsible for mentoring a team of data engineers, defining architectural best practices, ensuring the reliability of ingestion ecosystems, and collaborating closely with data platform, analytics, and business teams. **Key Responsibilities:**
Lead the design and development of enterprise-grade ingestion architectures using Databricks, SSIS, Azure Functions, and cloud services. - Drive modernization initiatives such as SSIS-to-ADF migration, batch-to-real-time ingestion, and automation improvements. - Ensure cost optimization, observability, data quality, and governance best practices. - Build and enhance ETL/ELT workflows using Python, Databricks (PySpark), SQL, and Azure components. - Develop C#/.NET services or functions for ingestion, validation, and integration with APIs/microservices. - Optimize ingestion workflows for performance, cost, and scalability. - Mentor and guide junior and mid-level data engineers. - Conduct architectural reviews, code reviews, and provide technical coaching. - Partner with Architects, DevOps Engineers, QA, and business stakeholders to align ingestion solutions with enterprise strategy. **Qualifications Required:**
Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field. - 710+ years of experience in data engineering/data ingestion, with at least 23 years in a lead role. - Deep experience with Databricks, Azure Functions, SSIS, and Python. - Strong programming background in Python and C#/.NET. - Solid hands-on experience with SQL Server, T-SQL, and performance tuning. - Experience building CI/CD pipelines with Azure DevOps. - Strong Python development skills for ingestion, automation, and transformation. - Experience migrating SSIS workloads to Azure-native tools. -