Key Responsibilities:
- Design and develop solutions using Bigdata tools and technologies like MapReduce, Hive, Spark etc.
- Extensive hands-on experience in object-oriented programming using Python, PySpark APIs etc.
- Experience in building data pipelines for huge volume of data.
- Experience in designing, implementing, and managing various ETL job execution flows.
- Experience in implementing and maintaining Data Ingestion process.
- Hands on experience in writing basic to advance level of optimized queries using HQL, SQL & Spark.
- Hands on experience in designing, implementing, and maintaining Data Transformation jobs using most efficient tools/technologies.
- Ensure the performance, quality, and responsiveness of solutions.
- Participate in code reviews to maintain code quality.
- Should be able to write shell scripts.
- Utilize Git for source version control.
- Set up and maintain CI/CD pipelines.
- Troubleshoot, debug, and upgrade existing application & ETL job chains.
Required Skills and Qualifications:
- Bachelor’s degree in Computer Science Engineering, or a related field.
- Proven experience as Data Engineer or similar role.
- Strong proficiency in Object Oriented programming using Python.
- Experience with ETL jobs design principles.
- Solid understanding of HQL, SQL and data modeling
- Knowledge on Unix/Linux and Shell scripting principles.
- Familiarity with Git and version control systems.
- Experience with Jenkins and CI/CD pipelines.
- Knowledge of software development best practices and design patterns.
- Excellent problem-solving skills and attention to detail.
- Strong communication and collaboration skills.
- Hands-on experience with Google Cloud.