Content Evaluator (Freelancer)

About Us

Deccan AI Experts is a pioneering AI company founded by IIT Bombay and IIM Ahmedabad alumni, with a strong founding team from IITs, NITs, and BITS. We specialize in high-quality human-curated data, AI-first operations, and advanced AI evaluation systems.

About the Role

We are seeking a Content Evaluator (Freelancer) to support advanced AI evaluation initiatives focused on content quality assessment, factual accuracy, instruction-following evaluation, safety review, and large language model (LLM) benchmarking.

In this role, you will review and evaluate AI-generated content across multiple formats, including text, documents, images, audio, video, and multimodal outputs. Your expertise will help improve next-generation AI systems by identifying quality issues, validating factual correctness, and providing structured feedback that enhances model performance.

This position is ideal for professionals with experience in content review, quality assurance, editorial review, AI evaluation, technical writing, research, journalism, education, or content moderation.

Responsibilities

• Evaluate AI-generated content across multiple modalities, including:
• Compare multiple AI-generated responses and rank them based on predefined quality guidelines.
• Identify factual errors, unsupported claims, inconsistencies, reasoning failures, formatting issues, and policy violations.
• Apply detailed evaluation rubrics to assign quality scores and document justification for evaluation decisions.
• Review AI-generated summaries, translations, classifications, question-answer pairs, creative writing, analytical reports, and conversational responses.
• Validate references, citations, structured outputs, and source-grounded content where applicable.
• Document edge cases, ambiguous outputs, and recurring model failure patterns to support continuous model improvement.
• Review peer evaluations and participate in calibration sessions to maintain high inter-rater reliability.
• Meet quality, consistency, and turnaround time expectations across assigned evaluation projects.

Requirements

• Bachelors degree in any discipline is preferred.
• 2+ years of experience in content evaluation, editorial review, quality assurance, research, journalism, AI evaluation, technical writing, education, content moderation, or similar analytical roles.
• Evaluating written content for quality, factual accuracy, clarity, and consistency.
• Applying structured evaluation guidelines and scoring rubrics.
• Identifying hallucinations, logical errors, unsupported claims, and instruction-following failures.
• Comparing multiple AI-generated responses objectively.
• Providing detailed written feedback supported by evidence.
• Excellent analytical thinking, critical reasoning, and attention to detail.
• Strong written English communication skills.
• Ability to make objective, consistent, and evidence-based evaluation decisions.
• Ability to work independently in a remote, fast-paced environment.

Preferred Qualifications

• Experience working with ChatGPT, Claude, Gemini, Copilot, Llama, DeepSeek, or other generative AI platforms.
• Experience in RLHF (Reinforcement Learning from Human Feedback), response ranking, preference evaluation, AI benchmarking, or prompt evaluation.
• Familiarity with content moderation, safety evaluation, policy enforcement, misinformation detection, or fact-checking workflows.
• Experience using annotation or evaluation platforms such as Label Studio, Scale AI, Labelbox, SuperAnnotate, Doccano, or similar tools.
• Background in journalism, research, law, education, consulting, healthcare, finance, or other knowledge-intensive domains is an advantage.
• Experience evaluating multimodal AI outputs, structured data, or agentic AI workflows is a plus.

Why Join Us

• Competitive hourly pay: ₹1,500/hour
• Fully remote with flexible working hours.
• Opportunity to contribute to cutting-edge AI evaluation and Generative AI initiatives.
• Exposure to advanced AI systems, benchmark development, and human feedback pipelines.
• Flexible project-based opportunities with global teams.
• Work on next-generation AI systems used across multiple industries and domains.