AI Engineer Retrieval & Evaluation

• AthenaPrep ResourcesOur StoryGuideBlogProfilePricing
• HelpdeskStart Free
• All open roles
• AIActively hiringPosted 6 weeks ago
• AI Engineer — Retrieval & Evaluation
• Make Ask Athena genuinely trustworthy. This is retrieval over our own question bank and a students own notes — where a confidently wrong answer costs someone marks.
• Apply for this role
• Experience 2–5 yrs
• Location Remote (India)
• Arrangement Remote
• Employment Full-time
• Department AI

Responsibilities

• Improve retrieval quality and answer grounding across our content corpus
• Build evaluation harnesses so answer quality is measured, not argued about
• Cut hallucination rates and make refusals honest rather than evasive
• Keep inference fast and cost-sane at student scale
• Work with Content to turn subject expertise into better grounding data

• 2+ years shipping ML or LLM features to real users
• Hands-on with embeddings, vector stores and retrieval pipelines
• Rigour about evaluation — you measure before and after, always
• Strong Python; comfortable in a TypeScript codebase

Nice to have, not required — apply even if none of these are true of you.

• Experience with Upstash Vector, pgvector or similar
• Background in education or assessment
• Published work, open-source contributions or writing on RAG

What success looks like

How well judge this role together — shared with you on day one, not at review time.

• Measurable reduction in ungrounded answers against a held-out eval set
• Median answer latency stays under two seconds
• Cost per assisted question trends down quarter over quarter

Growth path

• 1

First 90 days

Stand up the evaluation harness and establish the quality baseline.

• 2

Year one

Own the full AI surface — retrieval, grounding and cost.

• 3

  Beyond

• Define AI strategy as the products intelligence layer deepens.
• Skills youll use

• Python
• RAG
• Vector search
• LLM evaluation
• Prompt engineering

• Remote-first

Work from anywhere in India, with hours you actually control.

• Meaningful ownership

Early team members share in what they help build.

• Learning budget

Books, courses and conferences that make you better at the craft.

• Full AthenaPrep access

Every plan unlocked, for you and your immediate family.

• Equipment that helps

The machine and setup you need to do focused work.

• Time thats genuinely off

Generous leave, and no expectation of being reachable on it.

Hiring timeline

• 1

  You apply

• We reply within 3 days
• A short note on why this problem interests you beats a long CV. Links to things youve made help most.

• 2

  Intro conversation

• 30 minutes
• A relaxed call about your work, what you want next, and what were actually building. Questions go both ways.

• 3

  A real problem

• Take-home or live
• A small, time-boxed piece of work close to what the role actually involves. No trick puzzles, no unpaid project work.

• 4

  Conversation with the founder

• 45 minutes
• Values, ownership, and how youd want the first ninety days to go.

• 5

  Offer

• Within a week
• Clear terms, clear scope, and a start date that works for you.
• Questions
• Yes. We care much more about evidence of craft than years of experience — a project you built and can explain in depth counts for more than a familiar company name.
• Yes, remote-first across India, and async by default. We write decisions down so nobody needs to be online at a particular hour to stay in the loop.
• Usually under three weeks from application to offer. We reply to every application within three days, even when the answer is no.
• Yes, and they are real work with real ownership, not shadowing. Strong interns are regularly offered full-time roles.
• Join the talent community below. We open roles as we grow, and the people already in the pool are the first ones we contact.
• Next.js and TypeScript on the front end, Postgres via Supabase, and an AI layer built on retrieval over our own content. You do not need prior experience with all of it.
• Think this is you
• A short note on why this problem interests you beats a long CV. If youre close but not an exact match, apply anyway — wed rather read your note than miss you.
• Apply for AI Engineer — Retrieval & Evaluation Join talent community
• Apply now