Data Engineer (SQL)

As a Data Engineer at our company, you will be responsible for developing and maintaining data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python, and PySpark to ensure the efficient and reliable delivery of data. Your key responsibilities will include:

• Designing and implementing custom connectors to facilitate the ingestion of diverse data sources into our platform, including structured and unstructured data from various document formats. - Collaborating closely with cross-functional teams to gather requirements, understand data needs, and translate them into technical solutions. - Implementing DataOps principles and best practices to ensure robust data operations and efficient data delivery. - Designing and implementing data CI/CD pipelines to enable automated and efficient data integration, transformation, and deployment processes. - Monitoring and troubleshooting data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading. - Conducting data validation and testing to ensure the accuracy, consistency, and compliance of data. - Staying up-to-date with emerging technologies and best practices in data engineering. - Documenting data workflows, processes, and technical specifications to facilitate knowledge sharing and ensure data governance. Qualifications required for this role include:

• Bachelors degree in Computer Science, Engineering, or a related field. - 5+ years experience in data engineering, ELT development, and data modeling. - Proficiency in using Apache Airflow and Spark for data transformation, data integration, and data management. - Experience implementing workflow orchestration using tools like Apache Airflow, SSIS, or similar platforms. - Demonstrated experience in developing custom connectors for data ingestion from various sources. - Strong understanding of SQL and database concepts, with the ability to write efficient queries and optimize performance. - Experience implementing DataOps principles and practices, including data CI/CD pipelines. - Familiarity with data visualization tools Apache SuperSet and dashboard development. - Familiarity with data governance frameworks and practices. - Knowledge of data streaming and real-time data processing technologies. - Strong understanding of software development principles and practices, including version control. - Experience with Agile development methodologies and working in cross-functional Agile teams. - Strong written and verbal communication skills, with the ability to effectively communicate complex technical concepts to both technical and non-technical stakeholders. Additionally, you will be working in the Software Development department of our IT Services & Consulting industry. This is a full-time, permanent employment opportunity. As a Data Engineer at our company, you will be responsible for developing and maintaining data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python, and PySpark to ensure the efficient and reliable delivery of data. Your key responsibilities will include: