Sr Engineer - PLT - Engineering_PySpark

Experience: 3 to 7 Years Key Responsibilities . • Develop and maintain scalable data processing pipelines using PySpark, Scala, Python, and Apache Spark. • Build batch and near-real-time data processing solutions on Databricks. • Design, develop, and optimize ETL/ELT pipelines for large-scale data processing. • Write and optimize complex SQL queries, stored procedures, and data transformations. • Work with Delta Lake and Lakehouse architecture principles for efficient data management. • Develop reusable components and utilities for data ingestion, validation, and transformation. • Perform troubleshooting, debugging, and performance tuning of Spark applications and SQL workloads. • Ensure data quality, reliability, and adherence to engineering standards. Databricks & Cloud Development • Implement data solutions on Databricks using Spark and Delta Lake. • Work with AWS and/or Azure cloud services for data processing and storage. • Support deployment, monitoring, and maintenance of data pipelines. • Participate in performance optimization and cost optimization initiatives. Collaboration & Delivery • Collaborate with architects, senior engineers, QA teams, and business stakeholders to understand requirements. • Participate in sprint planning, estimation, development, testing, and release activities. • Contribute to code reviews and follow established coding standards and best practices. • Assist in resolving production issues and support activities. • Document technical designs, implementation details, and operational procedures. ________________________________________ Required Skills Technical Skills • Strong programming experience in Python, PySpark, or Scala. • Hands-on experience with Apache Spark (Core, SQL, DataFrames). • Experience working with Databricks and Delta Lake. • Strong proficiency in SQL and database development. • Experience building and maintaining ETL/ELT pipelines. • Hands-on experience with AWS and/or Azure cloud platforms. • Understanding of data lakes, data warehouses, and modern data platforms. • Knowledge of distributed computing and big data concepts. • Experience with data modeling and data processing best practices. Additional Skills • Knowledge of Core Java. • Experience writing Unit Tests and supporting code quality initiatives. • Familiarity with CI/CD pipelines and DevOps practices. • Experience using Git, Maven, Jira, and Agile methodologies. • Basic understanding of monitoring and logging frameworks. • Understanding of data security, governance, and compliance principles.