AI & Web Scraping-Python And Javascript (Remote)

Department Icon Data Science Analytics & Machine Learning
Posted: 2 months ago
3-5 years
Bengaluru / Bangalore, Karnataka
Data Science Analytics & Machine Learning
work from office
Full Time

Posted: 2 months ago
This job's closing date (9 Nov 2026) has passed, so the role may already be filled. See live similar jobs.
Let Jobaaj One apply for you

An AI agent that applies to roles like this on employers' own sites. 25 applications free, no card needed.

Job Description
About Company
Similar Jobs
Please verify your account first! Send OTP

Job Description

Python & JavaScript Developer – AI & Web Scraping

Experience: 3-5 Years 

Location: Remote 

Mode of Engagement: Full-time 

No of Positions: 3 

Educational Qualifications: Bachelors degree in computer science, Information Technology 

Industry: IT / Software Development 

Notice Period: Immediate Joiners Preferred

About the Role

We are looking for a hands-on Web Scraping / Crawling Engineer with 3–5 years of experience in web scraping, browser automation, and scalable data extraction.

The ideal candidate should have strong experience working with dynamic and JavaScript-heavy websites, building reliable crawling workflows, handling crawling failures, and working with distributed processing systems.

This is a highly technical, hands-on role. You will be expected to design, develop, debug, optimize, and maintain web crawling and data extraction systems.

Responsibilities

  • Design, develop, and maintain scalable web crawling and scraping systems for dynamic and JavaScript-heavy websites.
  • Develop browser automation workflows using Playwright, Selenium, Puppeteer, or similar frameworks.
  • Investigate and resolve crawling issues such as 403/429 responses, redirects, timeouts, rendering failures, session issues, and anti-bot challenges.
  • Develop robust retry, fallback, and failure-handling mechanisms to improve crawler reliability.
  • Work with cookies, sessions, browser contexts, headers, proxies, and related crawling mechanisms to maintain state and improve crawling reliability.
  • Build and optimize concurrent and distributed scraping workflows using asynchronous processing, queues, and worker-based architectures.
  • Design data pipelines covering URL processing, crawling, extraction, validation, transformation, and storage.
  • Optimize crawler performance, including concurrency, browser lifecycle, resource utilization, timeouts, and request handling.
  • Implement monitoring and observability for crawl success rates, failure types, latency, retries, and worker performance.
  • Debug complex crawling problems and identify root causes rather than relying only on one-off fixes.
  • Develop reusable crawling components and frameworks that can support multiple websites and use cases.
  • Collaborate with data engineering, AI, backend, and product teams to deliver reliable and structured web data.
  • Evaluate and adopt new web crawling, browser automation, and data extraction technologies where appropriate.

Required Skills

  • 3–5 years of hands-on experience in web scraping, web crawling, browser automation, or web data engineering.
  • Strong programming experience in Python. JavaScript/Node.js is a plus.
  • Strong hands-on experience with Scrapy, Playwright, Selenium, Puppeteer, or similar browser automation frameworks.
  • Strong understanding of HTTP, HTML, DOM, JavaScript rendering, redirects, cookies, sessions, headers, and browser contexts.
  • Experience with browser fingerprinting, WAFs, and modern anti-bot mechanisms.
  • Experience working with dynamic, JavaScript-heavy, and asynchronous websites.
  • Good understanding of asynchronous programming, concurrency, and parallel processing.
  • Experience with job queues, distributed workers, or message-based processing systems such as Redis, RabbitMQ, Kafka, Celery, or similar technologies.
  • Understanding of retry mechanisms, error handling, idempotency, rate limiting, and backpressure in distributed systems.
  • Hands-on experience with proxy management, session handling, and anti-bot challenges.
  • Strong debugging and analytical skills, with the ability to investigate and resolve complex crawling failures.
  • Experience working with SQL databases such as PostgreSQL or MySQL.
  • Familiarity with Docker and cloud platforms such as AWS, GCP, or Azure is a plus.
  • Strong hands-on engineering mindset rather than purely project or delivery management experience.
  • Ability to independently debug, investigate, and solve difficult crawling problems.
  • Strong understanding of how browsers, HTTP requests, sessions, and websites interact.
  • Ability to design systems that are reliable, scalable, and fault tolerant.
  • Good understanding of how to distribute and coordinate large-scale scraping workloads.
  • Ability to identify the root cause of crawling failures and develop generalized, reusable solutions.
  • Willingness to work across scraping, backend services, distributed processing, data pipelines, and infrastructure when required.
  • Strong problem-solving skills and curiosity to understand how websites behave rather than relying solely on existing scraping tools.

Education

  • Bachelors degree in Computer Science, Information Technology, Engineering, or a related field is preferred.
  • Equivalent practical experience in software engineering or web scraping will also be considered.

Skills

PythonData ExtractionMysqlAiSqlCelery

If a job posting appears fraudulent, asks for payment, contains misleading information, or violates our guidelines, please report it immediately. Our team will review it promptly, Jobaaj does not charge any fee from the applicants.

About Company

AIMLEAP is a trusted outsourcing partner for AI, data science, and big data solutions. We specialize in data annotation, data collection, data engineering, and custom AI model development. Our expertise helps businesses unlock the value of their data and gain a competitive edge.

Similar Jobs

View all
Loading...

Looking for something similar?

Bag Logo
Jobaaj
Get jobs like this by email

New Data Science Analytics & Machine Learning roles in Bengaluru / Bangalore.

Job Alert
Google hiring for Specific Roles Apply Now!
1 min ago
New Opportunity
Amazon is hiring freshers Apply Now!
5 min ago
Featured Jobs
Microsoft opening 50+ positions Apply Now!
10 min ago

One email a day. Unsubscribe any time.