Python Developer-LLM Evaluation & Validation

Location: Permanent Remote

Employment type: Contractor assignment (no medical/paid leave)

Duration of contract: 3 months

Commitment required:

• 20 40 hours/week with some overlap with PST
• First Priority 40 hrs/week with PST overlap (No dual employment)
• Second Priority 20 hrs/week with PST overlap (Part-time/dual employment allowed)

Must Have:

• 4+ years of relevant software development experience
• Python - min 3+ yrs of experience

What does day-to-day look like:

• Analyze and triage GitHub issues across trending open-source libraries.

• Set up and configure code repositories, including Dockerization and environment setup.

• Evaluating unit test coverage and quality.

• Modify and run codebases locally to assess LLM performance in bug-fixing scenarios.

• Collaborate with researchers to design and identify repositories and issues that are challenging for LLMs.

• Opportunities to lead a team of junior engineers to collaborate on projects.

Required Skills:

• Looking to get Placed? Try our Placement Guarantee Plan

  Get Hired
  Minimum 5+ years of overall experience

• Strong experience with at least one of the following languages: Python

• Proficiency with Git, Docker, and basic software pipeline setup.

• Ability to understand and navigate complex codebases.

• Comfortable running, modifying, and testing real-world projects locally.

• Experience contributing to or evaluating open-source projects is a plus.

Nice to Have:

• Previous participation in LLM research or evaluation projects.

• Experience building or testing developer tools or automation agents.