Senior Gen AI Data Engineer

Key Responsibilities

• Develop and maintain data ingestion pipelines using Azure Data Factory (ADF) and Databricks for structured and unstructured data.
• Create notebooks to process PDF and Word documents, including extracting text, tables, charts, graphs, and images.
• Apply NLP / Embedding Models (e.g., OpenAI, Hugging Face, sentence-transformers) to convert extracted content into embeddings.
• Store embeddings and metadata into Vector Databases (e.g., FAISS, Pinecone, Milvus, Weaviate, ChromaDB).
• Design and implement semantic search and retrieval workflows to enable prompt-based query capabilities.
• Optimize ETL pipelines for scalability, reliability, and performance.
• Collaborate with data scientists and solution architects to integrate GenAI capabilities into enterprise applications.
• 
  Follow best practices for code quality, modularity, and documentation.

Required Skills & Experience

• Proven experience in Azure Data Factory (ADF) and Databricks for building ETL/ELT

workflows.

• Strong programming experience in Python (pandas, PySpark, PyPDF, python-docx, OCR libraries, etc.).
• Hands-on experience with Vector Databases and semantic search implementation.
• Understanding of embedding models, LLM-based retrieval, and prompt engineering.
• Familiarity with handling multi-modal data (text, tables, images, charts).

  Looking to get Placed? Try our Placement Guarantee Plan

  Get Hired
• Strong knowledge of data modeling, indexing, and query optimization.
• Experience with cloud platforms (Azure preferred).
• Strong problem-solving, debugging, and communication skills.

Nice to Have

• Experience with knowledge graphs or RAG (Retrieval-Augmented Generation) pipelines.
• Exposure to MLOps practices and LLM fine-tuning.
• Familiarity with enterprise-scale document management systems.