Senior Consultant - Web Scraping

Department Icon IT / Software Development & Related
Posted: 1 week ago
8-12 years
Hyderabad
IT / Software Development & Related
work from office
Full Time
Closes 3 Jan 2027

Posted: 1 week ago
Let Jobaaj One apply for you

An AI agent that applies to roles like this on employers' own sites. 25 applications free, no card needed.

Job Description
About Company
Similar Jobs
Please verify your account first! Send OTP

Job Description

  • Senior Consultant - Web Scraping
  • Company: Chryselys
  • Job Location: Hyderabad, Telangana, India
  • Employment Type: Full-time
  • Experience Required: 8+ years
  • About Us:

Chryselys is a Great Place to Work Certified, Pharma Analytics & Business consulting company that delivers data-driven insights leveraging AI-powered, cloud-native platforms to achieve high-impact transformations.

  • We specialize in digital technologies and advanced data science techniques that provide strategic and operational insights.
  • Who we are:
  • People - Our team of industry veterans, advisors and senior strategists have diverse backgrounds and have worked at top tier companies.
  • Quality - Our goal is to deliver the value of a big five consulting company without the big five cost.
  • Technology - Our solutions are Business centric built on cloud native technologies.
  • Role summary:

Own and extend a production-bound POC that combines resilient web-crawling engineering with an AWS Bedrock-based LLM verification agent, in a healthcare-data-compliance-sensitive context.

  • ResponsibilitiesReplace scraped SERPs with a paid search API behind the existing search_domain() interface.
  • Move fetch.py to async httpx with per-domain concurrency limits and politeness budgets.
  • Introduce proxy rotation and egress management; retire the single-IP failure mode.
  • Make failure loud:
  • typed outcomes, structured logs, metrics, drift and volume alerts.
  • Reverse-engineer payer XHR/JSON endpoints to replace Playwright recipes wherever possible.
  • Honor robots.txt Disallow and Crawl-delay; build a per-domain ToS and licensing register.
  • Containerize and schedule the pipeline; add CI running the offline tests on every change.
  • Add HAR replay and golden-file parser tests on real payer HTML, plus daily canary crawls.
  • Experience - Required8+ years in data acquisition; 5+ years owning a scraping platform end to end.
  • Proven scale:
  • 1,000+ distinct domains or 10M+ pages/month sustained in production.
  • Rescued a brittle legacy scraper, with before/after reliability and cost numbers.
  • Mentored engineers; set crawl standards, review practice, and on-call runbooks.
  • Degree optional equivalent practical experience is fully accepted.
  • Nice-to-HaveUS payer policy, formulary, or prior-authorization document domain knowledge.

Open-source contributions to Scrapy, Playwright, Crawlee, or a parsing library.LLM-assisted extraction at controlled cost we run AWS Bedrock in verifier/.Compliance or legal-review exposure on data acquisition programmes.

  • Why Join Us
  • Work on Advanced Analytical techniques using AI/ML in the Market Access functions for the pharmaceutical domain with industry-leading tools.
  • Collaborate with Global and US teams to advance Market Access Contracting and Patient Analytics.
  • Opportunity to drive AI/ML-based innovations in Market Access Analytics.
  • Competitive salary, learning opportunities, and a dynamic work environment
  • Equal Employment Opportunity:
  • Chryselys is proud to be an Equal Employment Opportunity Employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.
  • Connect with Us:
  • Follow us for updates and more opportunities: https://linkedin.com/company/chryselys/mycompany
  • Discover more about our team and culture: http://chryselys.com .

Skills

Advanced Analytical TechniquesScraping PlatformProxy RotationDatadriven InsightsAIMLAWS BedrockWebcrawling EngineeringCIData Acquisition

If a job posting appears fraudulent, asks for payment, contains misleading information, or violates our guidelines, please report it immediately. Our team will review it promptly, Jobaaj does not charge any fee from the applicants.

About Company

Chryselys is a Pharma Analytics & Business consulting company that delivers data-driven insights leveraging AI-powered, cloud-native platforms to achieve high-impact transformations.We specialize in digital technologies and advanced data science techniques that provide strategic and operational insights.

Similar Jobs

View all
Loading...

Looking for something similar?

Bag Logo
Jobaaj
Get jobs like this by email

New IT / Software Development & Related roles in Hyderabad.

Job Alert
Google hiring for Specific Roles Apply Now!
1 min ago
New Opportunity
Amazon is hiring freshers Apply Now!
5 min ago
Featured Jobs
Microsoft opening 50+ positions Apply Now!
10 min ago

One email a day. Unsubscribe any time.