An AI agent that applies to roles like this on employers' own sites. 25 applications free, no card needed.
Job Description
- · Design and develop robust data ingestion and ETL pipelines for batch and workflow-based processing requirements.
- · Build, optimize, and maintain data processing solutions using Apache Spark, PySpark, and Apache NiFi.
- · Develop, schedule, monitor, and manage workflows in Apache Airflow to ensure reliable orchestration.
- · Work on the Cloudera Data Engineering platform and the broader Hadoop ecosystem, including HDFS and Hive.
- · Write efficient Hive SQL and Python code for data transformation, validation, and operational automation.
- · Support, troubleshoot, and optimize production data pipelines for performance, reliability, and scalability.
- · Collaborate with data architects, analysts, platform teams, and business stakeholders to understand requirements and deliver data solutions.
- · Follow engineering best practices for code quality, version control, documentation, deployment, and monitoring.
Work Experience
- · 8+ years of relevant experience in data engineering, big data platforms, and production pipeline delivery.
- · Strong hands-on experience in designing data pipelines using NiFi, Spark/PySpark, Hive SQL, and Python.
- · Good understanding of Hadoop concepts, HDFS storage patterns, data partitioning, and distributed processing.
- · Experience managing Airflow DAGs, dependencies, retries, monitoring, and failure handling.
- · Ability to identify pipeline bottlenecks and perform performance tuning across ingestion, transformation, and orchestration layers.
- Strong problem-solving mindset with the ability to support critical production workloads.
Skills
Apache SparkApache PysparkApache NifiHive SqlPythonApache AirflowHadoopApache HiveHadoop ClouderaHdfsSqlApache HadoopDistributed ProcessingData TransformationData ValidationIf a job posting appears fraudulent, asks for payment, contains misleading information, or violates our guidelines, please report it immediately. Our team will review it promptly, Jobaaj does not charge any fee from the applicants.
About Company
For more than 20 years, QuEST is a trusted global engineering services and solutions partner to many of the world's most recognized Fortune 500 companies in Aero Engines, Aerospace & Defense, Automotive, Medical Devices, Oil & Gas, Power, Industrial, Rail, Semiconductor, Consumer Electronics and Software & Internet. With global presence in 13 countries, 65 locations and 12000+ employees, QuEST is at the forefront of the convergence of Mechanical, Electronics, Software and Digital Services to engineer solutions for a safer, cleaner world. QuEST’s deep domain knowledge and digital expertise accelerate product development and innovation cycles, create alternate revenue streams, enhance consumer experience and make manufacturing processes and operations more efficient.
Similar Jobs
View all
New Data Science Analytics & Machine Learning roles in Yerwada.
One email a day. Unsubscribe any time.
