Airflow | Scheduling URLs in batches over multiple days

[devang pandey <de...@gmail.com>]

Hi ,

I need some suggestions related  to setting up scrapers with Airflow.

Issue is lets say i have a website to scrape that has  some 3k links. I
want to divide this over 3 batches of 1k each and all these have to run on
different days .

What can be the best approach in this case? If we can do some conditional
parameter basis scheduling. Example I have excel as my data source so next
to every url i can mention like Batch no ... now basis batch number if we
can schedule differently?

Hope this makes some sense. Please suggest .