This role is for Weekday's client.

Mandatory Skills: Python, PySpark, SQL, Azure Databricks, Azure DataFactory, Azure Synapse, ETL,Data Architect

Role Overview: Lead Data Engineer

We are looking for a Senior Data Engineer with a strong background in Python, Spark, Databricks, and Azure to join our growing team. The ideal candidate will have extensive experience in data processing, working with cloud platforms (specifically Azure), and be able to handle complex data engineering tasks. 

Key Responsibilities:

• Design, build, and optimize scalable data pipelines for large-scale data processing using Python and Apache Spark.
• Leverage Azure Databricks to perform big data processing, including working with PySpark to process and analyze large datasets.
• Utilize Azure Data Factory for seamless data integration and orchestration across cloud environments.
• Implement and maintain solutions using Azure Data Lake Storage for big data analytics and transformation.
• Write efficient and reusable Python code, utilizing libraries such as Pandas and NumPy, to process and manipulate data.
• Work with core Azure data services like Azure SQL Database, Azure Synapse Analytics, Azure Cosmos DB, and Azure Blob Storage.
• Design and maintain complex SQL queries for managing and analyzing relational databases.
• Ensure high-quality data management, integrity, and security across all stages of data processing and analysis.
• Collaborate with cross-functional teams to understand data requirements and build solutions to meet business needs.
• Provide mentorship and guidance to junior team members.

Key Skills & Qualifications:

• Experience: 7+ years of experience in data engineering or related roles, with a focus on cloud technologies and big data.
• Programming: Proficiency in Python, including knowledge of Pythonic principles, data structures, and algorithms. Hands-on experience with libraries such as Pandas and NumPy.
• Cloud Technologies: In-depth knowledge of Azure, including Azure SQL Database, Azure Synapse Analytics, Azure Cosmos DB, Azure Data Factory, Azure Blob Storage, and Azure Data Lake Storage.
• Data Processing: Strong experience with Apache Spark using Azure Databricks and PySpark for large-scale data processing.
• SQL: Strong skills in SQL for querying and managing relational databases. Ability to write complex and optimized SQL queries.
• Communication: Excellent communication skills, with the ability to collaborate effectively across teams and present technical solutions to non-technical stakeholders.
• Problem-Solving: Strong analytical and problem-solving skills, with the ability to troubleshoot and resolve complex data challenges