You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Ásgeir Halldórsson <as...@ec.is> on 2005/11/29 13:55:43 UTC
Setting up a crawler for a country.
Hello,
Is there anyone that can implement a country crawler? I estimate around 40m documents. Please send me info about your prev work and how much time it would take to setup and money :-)
Regards
Asgeir Halldorsson
Re: Setting up a crawler for a country.
Posted by "Insurance Squared Inc." <gc...@insurancesquared.com>.
Mostly an FYI post on those working with country specific SE's:
Just to continue on this topic, the country code TLD I'm looking at
doesn't provide any information so we're back to crawling to find
domains. To add to the complexity, there's lots of people who register
.com's as their main domain here, instead of the country specific TLD.
So our intended solution is to hack the filter so that it only crawls
and follows sites that match the specific TLD *or* match ARIN's IP list
for addresses in the country. (note: Arin publishes a list of IP
assignments by country). Not perfect, but it sure beats hand review.)
We'll assume that if they're hosted here that they're likely a site
relevant to the country.
For any remaining sites we're going to offer a manually submission
service. The sites will be reviewed manually, then added into the
filter. (we've got a preliminary php program that does this running now).
ARIN's IP mapping to country isn't quite perfect. For example our
servers are located here yet show up as being in the range of another
country. I expect I'll occassionally review the list of sites we've
added manually and look for trends in the IP address list to see if we
have any missing ranges. At that point I can pull those domains from
the filter and just add them to the IP address list.
I'm concerned that adding a huge range of IP's to check will cause the
crawler to slow. However of the four bytes in an ip address, there are
only about 10 possibilities in the first byte (i.e. the
000.XXX.XXX.XXX). So we'll check just the first byte, then continue to
drill down if there's a match.
HTH.
Matt Kangas wrote:
> glenn, i know that verisign makes this available for .com and .net as
> "TLD zone files".
>
> for ccTLDs like .us and .uk, you'll have to see if the TLD registrar
> provides the same. the following page has some useful links to these
> folks:
>
> http://www.dnsstuff.com/info/dnslinks.htm
>
> --matt
>
> On Nov 29, 2005, at 10:23 AM, Insurance Squared Inc. wrote:
>
>> Along these same lines (as I'm interested in a similiar country-
>> specific project), is there any place to get a list of all the
>> domains for a specific TLD to use to seed nutch? i.e. if I wanted
>> to get a list of all currently registered .it, .de, or .ca's?
>> I've looked without success. I'm thinking that this information
>> isn't available due to spamming issues, however in the paper you
>> referenced they discuss crawling an entire TLD which seemed to
>> indicate they may have access to this info.
>>
>> Thanks,
>> Glenn
>>
>>
>> Ken Krugler wrote:
>>
>>>
>>>> Is there anyone that can implement a country crawler? I
>>>> estimate around 40m documents. Please send me info about your
>>>> prev work and how much time it would take to setup and money :-)
>>>
>>>
>>>
>>> Check out the paper titled "Crawling a Country: Better Strategies
>>> than Breadth-First for Web Page Ordering" by Ricardo Baeza-Yates &
>>> others. They were using a crawl of Chilean domains to test
>>> strategies for efficient crawling, so it seems like it would be of
>>> interest to you.
>>>
>>> The main problem we've run into in doing similar limited domain
>>> crawls is that you wind up with many fewer hosts, and thus more
>>> URLs/host in any given fetch loop. The restriction of being polite
>>> (one thread per host) leads to lots of retry errors caused by
>>> fetcher threads blocking on a host (IP address) that is already
>>> being accessed by another fetcher thread, and thus lower pages/
>>> second throughput.
>>>
>>> So we've been making some mods to Nutch to improve our performance,
>>> but it's not debugged yet...getting closer, though.
>>>
>>> -- Ken
>>
>>
>
> --
> Matt Kangas / kangas@gmail.com
>
>
>
Re: Setting up a crawler for a country.
Posted by Matt Kangas <ka...@gmail.com>.
glenn, i know that verisign makes this available for .com and .net as
"TLD zone files".
for ccTLDs like .us and .uk, you'll have to see if the TLD registrar
provides the same. the following page has some useful links to these
folks:
http://www.dnsstuff.com/info/dnslinks.htm
--matt
On Nov 29, 2005, at 10:23 AM, Insurance Squared Inc. wrote:
> Along these same lines (as I'm interested in a similiar country-
> specific project), is there any place to get a list of all the
> domains for a specific TLD to use to seed nutch? i.e. if I wanted
> to get a list of all currently registered .it, .de, or .ca's?
> I've looked without success. I'm thinking that this information
> isn't available due to spamming issues, however in the paper you
> referenced they discuss crawling an entire TLD which seemed to
> indicate they may have access to this info.
>
> Thanks,
> Glenn
>
>
> Ken Krugler wrote:
>
>>
>>> Is there anyone that can implement a country crawler? I
>>> estimate around 40m documents. Please send me info about your
>>> prev work and how much time it would take to setup and money :-)
>>
>>
>> Check out the paper titled "Crawling a Country: Better Strategies
>> than Breadth-First for Web Page Ordering" by Ricardo Baeza-Yates &
>> others. They were using a crawl of Chilean domains to test
>> strategies for efficient crawling, so it seems like it would be of
>> interest to you.
>>
>> The main problem we've run into in doing similar limited domain
>> crawls is that you wind up with many fewer hosts, and thus more
>> URLs/host in any given fetch loop. The restriction of being polite
>> (one thread per host) leads to lots of retry errors caused by
>> fetcher threads blocking on a host (IP address) that is already
>> being accessed by another fetcher thread, and thus lower pages/
>> second throughput.
>>
>> So we've been making some mods to Nutch to improve our
>> performance, but it's not debugged yet...getting closer, though.
>>
>> -- Ken
>
--
Matt Kangas / kangas@gmail.com
Re: Setting up a crawler for a country.
Posted by "Insurance Squared Inc." <gc...@insurancesquared.com>.
Along these same lines (as I'm interested in a similiar country-specific
project), is there any place to get a list of all the domains for a
specific TLD to use to seed nutch? i.e. if I wanted to get a list of
all currently registered .it, .de, or .ca's?
I've looked without success. I'm thinking that this information isn't
available due to spamming issues, however in the paper you referenced
they discuss crawling an entire TLD which seemed to indicate they may
have access to this info.
Thanks,
Glenn
Ken Krugler wrote:
>
>> Is there anyone that can implement a country crawler? I estimate
>> around 40m documents. Please send me info about your prev work and
>> how much time it would take to setup and money :-)
>
>
> Check out the paper titled "Crawling a Country: Better Strategies than
> Breadth-First for Web Page Ordering" by Ricardo Baeza-Yates & others.
> They were using a crawl of Chilean domains to test strategies for
> efficient crawling, so it seems like it would be of interest to you.
>
> The main problem we've run into in doing similar limited domain crawls
> is that you wind up with many fewer hosts, and thus more URLs/host in
> any given fetch loop. The restriction of being polite (one thread per
> host) leads to lots of retry errors caused by fetcher threads blocking
> on a host (IP address) that is already being accessed by another
> fetcher thread, and thus lower pages/second throughput.
>
> So we've been making some mods to Nutch to improve our performance,
> but it's not debugged yet...getting closer, though.
>
> -- Ken
Re: Setting up a crawler for a country.
Posted by Ken Krugler <kk...@transpac.com>.
> Is there anyone that can implement a country crawler? I
>estimate around 40m documents. Please send me info about your prev
>work and how much time it would take to setup and money :-)
Check out the paper titled "Crawling a Country: Better Strategies
than Breadth-First for Web Page Ordering" by Ricardo Baeza-Yates &
others. They were using a crawl of Chilean domains to test strategies
for efficient crawling, so it seems like it would be of interest to
you.
The main problem we've run into in doing similar limited domain
crawls is that you wind up with many fewer hosts, and thus more
URLs/host in any given fetch loop. The restriction of being polite
(one thread per host) leads to lots of retry errors caused by fetcher
threads blocking on a host (IP address) that is already being
accessed by another fetcher thread, and thus lower pages/second
throughput.
So we've been making some mods to Nutch to improve our performance,
but it's not debugged yet...getting closer, though.
-- Ken
--
Ken Krugler
Krugle, Inc.
+1 530-470-9200