Data Engineer - Apache Spark

Job Title : Data EngineerKey Responsibilities :Data Pipeline Development :- Design and develop scalable data pipelines to process and transform large volumes of structured and unstructured data.Data Processing :- Build and optimize data processing frameworks using Apache Spark.Data Integration :- Integrate data from multiple sources to create reliable and efficient data workflows.Data Management :- Ensure efficient storage, retrieval, and transformation of data for analytics and reporting.System Performance :- Monitor and optimize data pipelines for performance, scalability, and reliability.Troubleshooting :- Identify and resolve issues in data pipelines and processing systems to ensure data accuracy and availability.Collaboration :- Work closely with data scientists, backend engineers, and product teams to understand data requirements and deliver solutions.Documentation :- Maintain clear documentation of data pipelines, data models, and workflows.Continuous Improvement :- Stay updated with emerging technologies and best practices in big data and data engineering.Required Skills :- Strong proficiency with Apache Spark and distributed data processing.- Experience designing and implementing scalable data pipelines.- Knowledge of data integration techniques across heterogeneous sources.- Ability to monitor, troubleshoot, and optimize data workflows for performance and reliability.- Good communication skills for crossfunctional collaboration and documentation.