Data Engineer - Web Scraping Specialist

The Opportunity

Were looking for a Data Engineer - Web Scraping Specialist (D3) to own and scale our data acquisition infrastructure. At the D3 level, youll design scalable data solutions, optimize our data platform, and enable data-driven decisions across the company.

"I design scalable data solutions, optimize our data platform, and enable data-driven decisions across the company."

The foundation is already in place weve built custom scrapers using NestJS, proxy services, and data pipelines and now we need a data engineer who can own, maintain, and scale these systems to handle serious growth. Youll build complex data pipelines that power our entire product ecosystem.

What Youll Do

As a Data Engineer (D3) specializing in web scraping, youll own significant parts of our data infrastructure, building complex acquisition pipelines and enabling analytics at scale.

Core D3 Responsibilities

• Design and implement complex data pipelines for web scraping and data acquisition
• Optimize data platform performance ensuring our scraping infrastructure scales smoothly
• Build real-time streaming solutions for continuous data collection and processing
• Implement data governance frameworks for scraped data quality and compliance
• Partner with product and engineering teams to prioritize data acquisition needs
• Lead data infrastructure projects that power business decisions
  
  

• Own and maintain existing scraper infrastructure built in NestJS
• Proactively monitor scraping targets and adapt to platform changes, rate limits, API behavior
• Implement proxy rotation strategies and anti-bot measures for operational continuity
• Design robust, resilient systems ready to handle millions of scraping targets
• Ensure data freshness and quality across all acquisition pipelines
  
  

• Backend: NestJS (TypeScript) for scraping services
• Data Processing: Real-time streaming and batch processing
• Storage: Data warehousing and optimization
• Monitoring: Pipeline observability and alerting
  
  

• Automation: Puppeteer, Playwright, Cheerio
• Infrastructure: Docker containerization, proxy services
• Anti-detection: Advanced evasion and rotation strategies
  
  

• Advanced data pipeline design you can architect scalable data acquisition systems
• Strong JavaScript/TypeScript skills particularly for browser automation and data processing
• Distributed computing experience with tools like Spark or similar frameworks
• Data modeling and architecture expertise for large-scale scraping operations
• Query optimization and performance tuning capabilities
  
  

• Expert-level experience with Puppeteer, Playwright, or Cheerio for web scraping
• Debugging and reverse-engineering sites using Chrome DevTools
• Proxy services and anti-bot measures implementation and management
• Complex scraping workflows at scale handling millions of targets
• Containerization experience (Docker) and microservices architecture
  
  

• Experience leading data infrastructure projects
• Ability to optimize platforms for 10x growth
• Strong collaboration with data scientists, analysts, and product teams
• Focus on data quality, governance, and compliance
  
  

• Fully onboarded to current scraping systems and data infrastructure
• Understanding of platforms we scrape and how data flows through our product
• Begin performing regular maintenance and optimization on scraper fleet
  
  

• Taking full ownership of scraping infrastructure and data pipelines
• Implementing structural improvements optimizations and scaling features
• Defining proxy handling strategies and anti-bot evasion at scale
• Building data governance frameworks for quality and compliance
  
  

• Leading complex data projects robust systems handling millions of targets
• Optimizing platform performance dramatic improvements in speed and cost
• Architecting new data solutions for additional platforms and use cases
• Recognized as data acquisition expert teams rely on your infrastructure
  
  

• Architect data platform components across multiple domains
• Lead multi-quarter data initiatives
• Influence data strategy company-wide
• Expand expertise beyond scraping into broader data engineering