Senior Data Engineer

What you will be doing, your legacy:

You will be working directly with company Principal Engineers evaluating, scoping, proposing, and building features to fulfill business solution requirements to protect our customers. You will play a direct role in laying the technical foundation for a new product offering. Additionally, you will be working with Engineering and DevOps to deliver high-quality products and services while also working closely with security and IT professionals to ensure safe and secure best practices are followed. Responsibilities:

• Architect and Design Scalable Data Solutions: Design/develop/maintain Data lakehouse solutions (Iceberg/Delta Lake /Hudi) applying industry best practices and structuring / optimizing the data according to data access patterns. - Data Pipeline Development: Implement ETL/ELT pipelines using cloud technologies (Spark / pySpark / Glue, Kinesis Streams / Iceberg) to load the data into a Lakehouse for both efficient ML processing and UI reporting. - Implement data models and data processing frameworks (Spark, Kafka, Snowflake) to ingest, transform, and load large datasets into Data Lakehouse techs (Apache Iceberg, Apache Delta Lake or Apache Hudi), ensuring high availability and reliability of data. - Advanced Data Integration: Develop solutions that integrate multiple data sources into Snowflake or similar data warehouses to enable real-time analytics and reporting across dashboards. - AI/ML Integration: Collaborate with cross-functional teams to co-develop AI-driven features identifying patterns and anomalies in client data using AI/ML technologies (python). - Ensure compliance with industry standards and secure best practices (SOX, SOC 1/2), by implementing data governance frameworks, monitoring data pipelines, and optimizing cloud database architectures to protect sensitive information. - Work closely with stakeholders, including analysts, engineers, and product managers, to understand their data needs, propose solutions, and drive data-driven decision-making by delivering actionable insights. - Continuously monitor, troubleshoot, and enhance data pipelines, leveraging CI/CD tools (Docker, Jenkins, GitHub Actions) and orchestrating workflows using Apache Airflow to maintain operational efficiency. - Provide hands-on mentorship and technical guidance to junior engineers, including code reviews and architecture discussions. - Establish comprehensive documentation for data architecture, governance, and processes to ensure scalability, compliance, and security. ###