Data Engineer

Cloud, Mobile, Cognitive, Security, Research, Watson, Analytics, Consulting, Commerce, Experience Design, Internet of Things, Technology support, Industry solutions, Systems services, Resiliency services, Financing, and IT infrastructure

Mandatory Skills: Spark, Scala,Python,Java, SQL queries , Kafka

Position : Contract to hire

Location: Bangalore

Experience: 4- 9 Yrs

Job Description:

Sr. Data Engineer

Summary

• Executes the development, maintenance, and enhancements of data ingestion solutions of varying complexity levels across various data sources like DBMS, File systems (structured and unstructured), APIs and Streaming on on-prem and cloud infrastructure; demonstrates strong acumen in Data Ingestion toolsets and nurtures and grows junior members in this capability
• Builds, tests and enhances data curation pipelines integration data from wide variety of sources like DBMS, File systems, APIs and streaming systems for various KPIs and metrics development with high data quality and integrity
• Supports the development of feature / inputs for the data models in an Agile manner; Hosts Model Via Rest APIs; ensures non-functional requirements such as logging, authentication, error capturing, and concurrency management are accounted for when model hosting
• Works with Data Science team to understand mathematical models and algorithms; recommends improvements to analytic methods, techniques, standards, policies and procedures
• Works to ensure the manipulation and administration of data and systems are secure and in accordance with enterprise governance by staying compliant with industry best practices, enterprise standards, corporate policy and department procedures; handles the manipulation (extract, load, transform), data visualization, and administration of data and systems securely and in accordance with enterprise data governance standards
• Maintains the health and monitoring of assigned data engineering capabilities that span analytic functions by triaging maintenance issues; serves as an SME of one or more application

Qualifications

Minimum Qualifications

• Bachelor's Degree in Engineering, Computer Science, CIS, or related field (or equivalent work experience in a related field)

• 6 years of progressive experience in Data, Software Engineering (preferably started with Software Engineering and transitioned to Data)

Preferred Qualifications

• Master's Degree in Computer Science, CIS, or related field

• 6-8 years of experience in Hadoop, NO-SQL(at least 5 years exp), RDBMS or any Cloud Big Data components

• 3 years of experience in systems analysis, including defining technical requirements and performing high level design for complex solutions

• 3 years of experience writing technical documentation in a software development environment

• 5 years of experience working with source code control systems

• Experience working with Continuous Integration/Continuous Deployment tools 

• Expertise in Java/Scala, SQL, Scripting, Hadoop (Sqoop, Hive, Pig, Map Reduce), Spark (SQL, Spark Streaming, MLib), equivalent Cloud Big Data components

• Experience in Streaming integration of Kafka using Kafka client/connectors (using Spring boot or similar)

• Experience in building restful API's(Microservices) to expose Data using frameworks like Spring boot.