Skip to content

Wayfarer

Ruby web crawling framework built on ActiveJob and Redis

Read the tutorial 

gem install wayfarer

gem "wayfarer"
  • Breadth-first, acyclic page traversal
  • Plain HTTP and browser automation via Ferrum (CDP), Selenium and custom user agents
  • Declarative routing DSL
  • URI normalization and deduplication
  • HTML, XML, JSON and custom Content-Type body parsing