Job Overview

We are seeking a skilled Data Engineer for a full-time, mid-level position in Hyderabad or Bangalore. Candidates should bring between 4 to 6 years of relevant work experience to this role, where they will leverage their expertise in data management and optimization to further our company's goals in the HR industry.

Qualifications and Skills

• Expertise in SQL, Python, ETL Tools to enable efficient data processing and transformation (Mandatory skill).
• Proven experience with Apache Spark to perform large-scale data processing and analysis.
• Hands-on experience with Kafka for building real-time data streaming applications.
• Strong understanding of Data Warehousing solutions and the ability to design scalable and effective warehouse environments.
• Hands-on experience with Apache Airflow for orchestrating complex data workflows.
• Experience working with Azure cloud services, particularly in a data engineering context, to implement cloud-based data solutions.
• Ability to collaborate effectively with cross-functional teams to understand data needs and deliver solutions that meet business requirements.
• Excellent problem-solving skills and the ability to troubleshoot database and data pipeline performance issues.

Roles and Responsibilities

• Design, implement, and maintain scalable data pipelines that facilitate efficient data processing and analysis.
• Develop and optimize SQL queries and scripts for data extraction, transformation, and loading purposes.
• Collaborate with data analysts and scientists to ensure data solutions align with analytical needs.
• Implement data quality checks and procedures to ensure the integrity and accuracy of data across platforms.
• Monitor system performance and make required adjustments to optimize data flow and resource allocation.
• Use Apache Spark and other tools to manage and process large datasets efficiently.
• Contribute to the improvement of existing data systems and recommend innovative approaches for data management.
• Provide technical guidance and support to junior engineers and cross-functional teams as needed.