Senior Data Engineer

At Crossing Hurdles, we work as a referral partner. We refer candidates to our partner that collaborates with worlds leading AI research labs to build and train cutting-edge AI models.

Position: Senior Data Engineer

Type: Hourly Contract

Compensation: $14 per hour

Location: India (Remote)

Commitment: Part-time (20-30 hours/week), flexible

Role Responsibilities:

• Design and maintain scalable ETL/ELT pipelines that ensure datasets are clean, reliable, and ML-ready.
• Validate, enrich, and transform datasets across diverse formats including JSON, CSV, and Parquet.
• Manage data contracts, schema evolution, and version control to maintain consistency across pipelines.
• Work with PostgreSQL/SQLite and distributed processing systems like Spark or DuckDB.
• Orchestrate workflows using tools such as Airflow or equivalent schedulers.
• Optimize data pipelines for performance and reliability using Python and pandas.
• Collaborate with researchers and engineers to align data infrastructure with analytics and AI research needs.

Requirements:

• Strong background in data engineering, computer science, or information systems.
• Proficient in Python, pandas, and SQL.
• Hands-on experience with relational databases such as PostgreSQL or SQLite.
• Working knowledge of distributed data processing (Spark or DuckDB).
• Experience with workflow orchestration tools like Airflow.
• Familiarity with common data formats (JSON, CSV, Parquet).
• Strong understanding of schema design, data contracts, and version control using Git.
• Passion for building reliable, reproducible, and scalable data pipelines that power analytics and ML systems.

Key Domains:If you specialize in any of the following, you are encouraged to apply:

• ETL/ELT Pipelines: Data transformations, enrichment, validation
• Data Engineering: Workflow orchestration, distributed processing, schema management
• Database Systems: PostgreSQL, SQLite, query optimization
• AI & ML Data Support: ML-ready datasets, reproducible dataset creation
• Data Infrastructure: Airflow, Spark, DuckDB, Parquet pipelines

Application Process:

• Apply for the job role
• Await the official message/email from our recruitment team (typically within 12 days)