Redis

Wayfarer uses Redis to keep track of:

• URLs that were already processed within a batch
• the number of jobs left in a batch

Garbage collection

Wayfarer cleans up batch-related data