You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Krish Pan <ce...@gmail.com> on 2010/10/27 23:29:02 UTC
downloading exact number of pages from list of seed urls
Hi,
I am trying to use nutch to just download exact number of (say 5) html pages
from each seed page I provide,
I was wondering if this is the right approach,
Seed Urls = total 10,000
bin/nuch crawl urls/<list of domains> -dir <out> -depth 1 -topN 60000
here depth = 1 because I just want pages from only first level
i.e. if domain if foo.bar I want to download
foo.bar/spam.htm
foo.bar/ham.htm
foo.bar/eggs.htm
but NOT
foo.bar/ham/spam.htm
And,
-topN is 60,000 because there there are 10,000 seed urls
10,000 home pages and 5 top pages per url
Any suggestions?
Thanks,
krish
Re: downloading exact number of pages from list of seed urls
Posted by Krish Pan <ce...@gmail.com>.
Rob,
that helps a lot,
Is this related to limiting urls/host in any way?
https://issues.apache.org/jira/browse/NUTCH-272
<https://issues.apache.org/jira/browse/NUTCH-272>Thanks,
Krish
On Wed, Oct 27, 2010 at 5:14 PM, Rob Hunter <rh...@oversee.net> wrote:
> Krish,
>
> I think what you're looking for is a depth of 2 - I believe depth of
> 1 will only return foo.bar. Also, due to your depth change, I think you
> can reduce your topN to 50k. I'm unsure if your results will be evenly
> distributed across your domains, hopefully someone else has an answer
> for that.
>
> -- Rob
>
> -----Original Message-----
> From: Krish Pan [mailto:ceo.coder@gmail.com]
> Sent: Wednesday, October 27, 2010 2:29 PM
> To: user@nutch.apache.org
> Subject: downloading exact number of pages from list of seed urls
>
> Hi,
>
> I am trying to use nutch to just download exact number of (say 5) html
> pages
> from each seed page I provide,
>
> I was wondering if this is the right approach,
>
> Seed Urls = total 10,000
>
> bin/nuch crawl urls/<list of domains> -dir <out> -depth 1 -topN 60000
>
> here depth = 1 because I just want pages from only first level
>
> i.e. if domain if foo.bar I want to download
>
> foo.bar/spam.htm
> foo.bar/ham.htm
> foo.bar/eggs.htm
>
> but NOT
> foo.bar/ham/spam.htm
>
> And,
>
> -topN is 60,000 because there there are 10,000 seed urls
> 10,000 home pages and 5 top pages per url
>
> Any suggestions?
>
> Thanks,
> krish
>
Re: downloading exact number of pages from list of seed urls
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 2010-10-28 19:58, Rob Hunter wrote:
> Andrzej, do you know if the results will be evenly distributed? In
> that setup, will there be five depth 2 sites returned from each seed
> url, or could it end up being 100 from the first and none from the
> last 20?
There's no guarantee they will be evenly distributed - this depends on
the relative "importance" (score) of the pages. However, you can limit
the max. number of pages per host that you want to have in a fetchlist,
which helps to balance the fetching process.
>
> There's also a property called db.max.outlinks.per.page - can that be
> used to limit the number of depth 2 sites fetched?
Yes, though that's not the primary purpose of this property - it's
rather to limit resource consumption during updatedb operation, the size
of the linkdb, etc. But the secondary purpose is indeed to limit an
undue impact of pages with many outlinks, so that they don't overwhelm
crawldb with their links.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
RE: downloading exact number of pages from list of seed urls
Posted by Rob Hunter <rh...@oversee.net>.
Andrzej, do you know if the results will be evenly distributed? In that setup, will there be five depth 2 sites returned from each seed url, or could it end up being 100 from the first and none from the last 20?
There's also a property called db.max.outlinks.per.page - can that be used to limit the number of depth 2 sites fetched?
-- Rob
-----Original Message-----
From: Andrzej Bialecki [mailto:ab@getopt.org]
Sent: Thursday, October 28, 2010 1:43 AM
To: user@nutch.apache.org
Subject: Re: downloading exact number of pages from list of seed urls
On 2010-10-28 02:14, Rob Hunter wrote:
> Krish,
>
> I think what you're looking for is a depth of 2 - I believe depth of
> 1 will only return foo.bar. Also, due to your depth change, I think you
> can reduce your topN to 50k. I'm unsure if your results will be evenly
> distributed across your domains, hopefully someone else has an answer
> for that.
Actually, the "depth" is a misnomer... it doesn't relate to the actual
depth of URL paths, only to the number of hops from the seed pages. A
seed page www.a.com/index.html may contain links like this:
www.a.com/one.html
www.a.com/two/one.html
www.a.com/two/three/one.html
..and from the point of view of Nutch they are all at "depth" 2, i.e. it
takes two rounds of fetching to fetch them - the first round to fetch
the see page, and the second round to fetch the linked pages.
In other words, depth=1 means "fetch the seed pages", depth=2 means
"fetch pages outlinked from seed pages", and so on...
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: downloading exact number of pages from list of seed urls
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 2010-10-28 02:14, Rob Hunter wrote:
> Krish,
>
> I think what you're looking for is a depth of 2 - I believe depth of
> 1 will only return foo.bar. Also, due to your depth change, I think you
> can reduce your topN to 50k. I'm unsure if your results will be evenly
> distributed across your domains, hopefully someone else has an answer
> for that.
Actually, the "depth" is a misnomer... it doesn't relate to the actual
depth of URL paths, only to the number of hops from the seed pages. A
seed page www.a.com/index.html may contain links like this:
www.a.com/one.html
www.a.com/two/one.html
www.a.com/two/three/one.html
..and from the point of view of Nutch they are all at "depth" 2, i.e. it
takes two rounds of fetching to fetch them - the first round to fetch
the see page, and the second round to fetch the linked pages.
In other words, depth=1 means "fetch the seed pages", depth=2 means
"fetch pages outlinked from seed pages", and so on...
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
RE: downloading exact number of pages from list of seed urls
Posted by Rob Hunter <rh...@oversee.net>.
Krish,
I think what you're looking for is a depth of 2 - I believe depth of
1 will only return foo.bar. Also, due to your depth change, I think you
can reduce your topN to 50k. I'm unsure if your results will be evenly
distributed across your domains, hopefully someone else has an answer
for that.
-- Rob
-----Original Message-----
From: Krish Pan [mailto:ceo.coder@gmail.com]
Sent: Wednesday, October 27, 2010 2:29 PM
To: user@nutch.apache.org
Subject: downloading exact number of pages from list of seed urls
Hi,
I am trying to use nutch to just download exact number of (say 5) html
pages
from each seed page I provide,
I was wondering if this is the right approach,
Seed Urls = total 10,000
bin/nuch crawl urls/<list of domains> -dir <out> -depth 1 -topN 60000
here depth = 1 because I just want pages from only first level
i.e. if domain if foo.bar I want to download
foo.bar/spam.htm
foo.bar/ham.htm
foo.bar/eggs.htm
but NOT
foo.bar/ham/spam.htm
And,
-topN is 60,000 because there there are 10,000 seed urls
10,000 home pages and 5 top pages per url
Any suggestions?
Thanks,
krish