Crawling the entire web

[Ridwan Naibi <ri...@gmail.com>]

Hi everyone,

I hope this meets you well. I am planning to crawl the entire web. I already know how to setup nutch, solr, and a database. I need advice on how to crawl the entire web. How many nodes should I have? What instances should I use on say AWS Cloud? What should my setup be like? This may sound like I am asking to be spoon-fed but I am hoping someone has already done same so that I can get good advice. I want to get all active links on the web for research purposes. Google APIs have a limitation of 100 links. Accessing zone files have been a Herculean task because no single registrar seems to have them all not at a prohibitive cost. And I want to get weekly updates or monthly of all links. I am open all suggestions.

Best
Ridwan

Re: Crawling the entire web

[Ridwan Naibi <ri...@gmail.com>]

Hi Gora,

Thank you very much for your advice. That is exactly what I need.

Best wishes
Ridwan
________________________________
From: Gora Mohanty <go...@mimirtech.com>
Sent: Wednesday, January 10, 2024 5:21:49 PM
To: user@nutch.apache.org <us...@nutch.apache.org>
Subject: Re: Crawling the entire web

Hi,

Would suggest starting out by looking at Common Crawl:
https://commoncrawl.org/

Regards,
Gora

Re: Crawling the entire web

[Gora Mohanty <go...@mimirtech.com>]

Hi,

Would suggest starting out by looking at Common Crawl:
https://commoncrawl.org/

Regards,
Gora