You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Feng (Michael) Ji" <fj...@yahoo.com> on 2005/07/24 16:03:33 UTC
fetching behavior of Nutch
hi there,
1)
I did several testing running to fetch page from two
website. The fetching depth is 10.
After checking log files, I found the actual fetched
page linkage is very different for two sites.
In one site with lots of news, only first two depth
fetching running well and only fetching 5 linkages.
The actual linkages in that site is far beyond that.
The other site can fetch till 10 rounds and fetched
100's linkage.
I wonder if any one has similar experience. Should I
setup configure files in /conf/?
2)
Also, in Nutch/conf/ directory, I found several
configuration files. Actually, I only modify
crawl-urlfilter.txt to let it accept all the url
(*.*).
Is it proper?
I really doesn't touch other conf files. Is there a
guideline how I use these files?
thanks,
Michael,
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Re: Nutch's intranet VS internet crawling
Posted by Jack Tang <hi...@gmail.com>.
Hi Michael
I think some information in intranet crawling is privacy while
internet is public.
/Jack
On 7/24/05, Feng (Michael) Ji <fj...@yahoo.com> wrote:
>
> I wonder if there is any difference between these two?
> Or intranet crawling must indicate an intranet site
> explicitly in crawl-urlfilter.txt under /conf ?
>
> thanks,
>
> Michael,
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
--
Keep Discovering ... ...
http://www.jroller.com/page/jmars
RE: fetching behavior of Nutch
Posted by "Feng (Michael) Ji" <fj...@yahoo.com>.
thanks Howie,
that guides me,
Michael,
--- Howie Wang <ho...@hotmail.com> wrote:
> There are probably two settings you'll need to tweak
> in nutch-default.xml
>
> http.content.limit -- by default it's 64K, if the
> page is
> larger than that, then it essentially truncates the
> file.
> You could be missing lots of links that appear later
> in
> the page.
>
> max.outlinks.per.page -- by default it's 100. You
> might
> want to increase this since for pages with something
> like
> a nested navigation sidebar with tons of links, it
> won't
> get any links from the main part of the page.
>
> The *.xml files are fairly descriptive. So just
> reading through
> them can be pretty helpful. I don't know if there is
> a full
> guide to the config files.
>
> Howie
>
>
>
> >
> >1)
> >I did several testing running to fetch page from
> two
> >website. The fetching depth is 10.
> >
> >After checking log files, I found the actual
> fetched
> >page linkage is very different for two sites.
> >
> >In one site with lots of news, only first two depth
> >fetching running well and only fetching 5 linkages.
> >The actual linkages in that site is far beyond
> that.
> >
> >The other site can fetch till 10 rounds and fetched
> >100's linkage.
> >
> >I wonder if any one has similar experience. Should
> I
> >setup configure files in /conf/?
> >
> >2)
> >Also, in Nutch/conf/ directory, I found several
> >configuration files. Actually, I only modify
> >crawl-urlfilter.txt to let it accept all the url
> >(*.*).
> >
> >Is it proper?
> >
> >I really doesn't touch other conf files. Is there a
> >guideline how I use these files?
> >
> >thanks,
> >
> >Michael,
> >
> >
> >
> >__________________________________________________
> >Do You Yahoo!?
> >Tired of spam? Yahoo! Mail has the best spam
> protection around
> >http://mail.yahoo.com
>
>
>
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
RE: fetching behavior of Nutch
Posted by "Feng (Michael) Ji" <fj...@yahoo.com>.
Hi Howie:
Your direction solve the problem. You are right, the
site I crawled has huge page size. Thanks.
Saying, the one site having a page built by pointing
to company.com/G1/I273/;
I checked log file and Nutch doesn't fetch that
structure;
and in search engine side, when I type the content of
that particular page, it returns NULL.
I wonder if it is due to Nutch's crawler can't fetch
that page structure.
thanks,
Michael,
--- Howie Wang <ho...@hotmail.com> wrote:
> There are probably two settings you'll need to tweak
> in nutch-default.xml
>
> http.content.limit -- by default it's 64K, if the
> page is
> larger than that, then it essentially truncates the
> file.
> You could be missing lots of links that appear later
> in
> the page.
>
> max.outlinks.per.page -- by default it's 100. You
> might
> want to increase this since for pages with something
> like
> a nested navigation sidebar with tons of links, it
> won't
> get any links from the main part of the page.
>
> The *.xml files are fairly descriptive. So just
> reading through
> them can be pretty helpful. I don't know if there is
> a full
> guide to the config files.
>
> Howie
>
>
>
> >
> >1)
> >I did several testing running to fetch page from
> two
> >website. The fetching depth is 10.
> >
> >After checking log files, I found the actual
> fetched
> >page linkage is very different for two sites.
> >
> >In one site with lots of news, only first two depth
> >fetching running well and only fetching 5 linkages.
> >The actual linkages in that site is far beyond
> that.
> >
> >The other site can fetch till 10 rounds and fetched
> >100's linkage.
> >
> >I wonder if any one has similar experience. Should
> I
> >setup configure files in /conf/?
> >
> >2)
> >Also, in Nutch/conf/ directory, I found several
> >configuration files. Actually, I only modify
> >crawl-urlfilter.txt to let it accept all the url
> >(*.*).
> >
> >Is it proper?
> >
> >I really doesn't touch other conf files. Is there a
> >guideline how I use these files?
> >
> >thanks,
> >
> >Michael,
> >
> >
> >
> >__________________________________________________
> >Do You Yahoo!?
> >Tired of spam? Yahoo! Mail has the best spam
> protection around
> >http://mail.yahoo.com
>
>
>
__________________________________
Yahoo! Mail for Mobile
Take Yahoo! Mail with you! Check email on your mobile phone.
http://mobile.yahoo.com/learn/mail
RE: fetching behavior of Nutch
Posted by "Feng (Michael) Ji" <fj...@yahoo.com>.
thanks Howie,
that guides me,
Michael,
--- Howie Wang <ho...@hotmail.com> wrote:
> There are probably two settings you'll need to tweak
> in nutch-default.xml
>
> http.content.limit -- by default it's 64K, if the
> page is
> larger than that, then it essentially truncates the
> file.
> You could be missing lots of links that appear later
> in
> the page.
>
> max.outlinks.per.page -- by default it's 100. You
> might
> want to increase this since for pages with something
> like
> a nested navigation sidebar with tons of links, it
> won't
> get any links from the main part of the page.
>
> The *.xml files are fairly descriptive. So just
> reading through
> them can be pretty helpful. I don't know if there is
> a full
> guide to the config files.
>
> Howie
>
>
>
> >
> >1)
> >I did several testing running to fetch page from
> two
> >website. The fetching depth is 10.
> >
> >After checking log files, I found the actual
> fetched
> >page linkage is very different for two sites.
> >
> >In one site with lots of news, only first two depth
> >fetching running well and only fetching 5 linkages.
> >The actual linkages in that site is far beyond
> that.
> >
> >The other site can fetch till 10 rounds and fetched
> >100's linkage.
> >
> >I wonder if any one has similar experience. Should
> I
> >setup configure files in /conf/?
> >
> >2)
> >Also, in Nutch/conf/ directory, I found several
> >configuration files. Actually, I only modify
> >crawl-urlfilter.txt to let it accept all the url
> >(*.*).
> >
> >Is it proper?
> >
> >I really doesn't touch other conf files. Is there a
> >guideline how I use these files?
> >
> >thanks,
> >
> >Michael,
> >
> >
> >
> >__________________________________________________
> >Do You Yahoo!?
> >Tired of spam? Yahoo! Mail has the best spam
> protection around
> >http://mail.yahoo.com
>
>
>
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
RE: fetching behavior of Nutch
Posted by Howie Wang <ho...@hotmail.com>.
There are probably two settings you'll need to tweak
in nutch-default.xml
http.content.limit -- by default it's 64K, if the page is
larger than that, then it essentially truncates the file.
You could be missing lots of links that appear later in
the page.
max.outlinks.per.page -- by default it's 100. You might
want to increase this since for pages with something like
a nested navigation sidebar with tons of links, it won't
get any links from the main part of the page.
The *.xml files are fairly descriptive. So just reading through
them can be pretty helpful. I don't know if there is a full
guide to the config files.
Howie
>
>1)
>I did several testing running to fetch page from two
>website. The fetching depth is 10.
>
>After checking log files, I found the actual fetched
>page linkage is very different for two sites.
>
>In one site with lots of news, only first two depth
>fetching running well and only fetching 5 linkages.
>The actual linkages in that site is far beyond that.
>
>The other site can fetch till 10 rounds and fetched
>100's linkage.
>
>I wonder if any one has similar experience. Should I
>setup configure files in /conf/?
>
>2)
>Also, in Nutch/conf/ directory, I found several
>configuration files. Actually, I only modify
>crawl-urlfilter.txt to let it accept all the url
>(*.*).
>
>Is it proper?
>
>I really doesn't touch other conf files. Is there a
>guideline how I use these files?
>
>thanks,
>
>Michael,
>
>
>
>__________________________________________________
>Do You Yahoo!?
>Tired of spam? Yahoo! Mail has the best spam protection around
>http://mail.yahoo.com
fetcher blocked
Posted by EM <em...@cpuedge.com>.
I'm using 0.7 from a few weeks ago.
I was fetching 204 456 pages,
Nutch -segread tells that there are:
"segments\20050723140812 is corrupt, using only 207126 entries."
Here's what I have with ctrl-break:
Full thread dump Java HotSpot(TM) Client VM (1.4.2_08-b03 mixed mode):
"MultiThreadedHttpConnectionManager cleanup" daemon prio=5 tid=0x032605e8
nid=0x1034 in Object.wait() [3b0f000..3b0fd8c]
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x150622f0> (a java.lang.ref.ReferenceQueue$Lock)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQu
eueThread.run(MultiThreadedHttpConnectionManager.java:1100)
"fetcher8" prio=5 tid=0x03230888 nid=0x1d04 in Object.wait()
[354f000..354fd8c]
at java.lang.Object.wait(Native Method)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnec
tion(MultiThreadedHttpConnectionManager.java:509)
- locked <0x15063090> (a
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionP
ool)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnecti
onWithTimeout(MultiThreadedHttpConnectionManager.java:394)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDir
ector.java:152)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:393)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:324)
at
org.apache.nutch.protocol.httpclient.HttpResponse.<init>(HttpResponse.java:7
6)
at
org.apache.nutch.protocol.httpclient.Http.getProtocolOutput(Http.java:213)
at
org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:135)
"Signal Dispatcher" daemon prio=10 tid=0x009ff8a8 nid=0xfdc waiting on
condition [0..0]
"Finalizer" daemon prio=9 tid=0x009fcd20 nid=0x900 in Object.wait()
[2f7f000..2f7fd8c]
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
- locked <0x14ee5b08> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x009fb9a0 nid=0xeb4 in Object.wait()
[2f3f000..2f3fd8c]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:115)
- locked <0x14ee5b70> (a java.lang.ref.Reference$Lock)
"main" prio=5 tid=0x000366e0 nid=0x10d0 waiting on condition [7f000..7fc38]
at java.lang.Thread.sleep(Native Method)
at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:342)
at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:479)
"VM Thread" prio=5 tid=0x00a3b720 nid=0x1230 runnable
"VM Periodic Task Thread" prio=10 tid=0x00a3d360 nid=0xae0 waiting on
condition
"Suspend Checker Thread" prio=10 tid=0x009feeb0 nid=0x27c runnable
Nutch's intranet VS internet crawling
Posted by "Feng (Michael) Ji" <fj...@yahoo.com>.
I wonder if there is any difference between these two?
Or intranet crawling must indicate an intranet site
explicitly in crawl-urlfilter.txt under /conf ?
thanks,
Michael,
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
RE: fetching behavior of Nutch
Posted by Howie Wang <ho...@hotmail.com>.
There are probably two settings you'll need to tweak
in nutch-default.xml
http.content.limit -- by default it's 64K, if the page is
larger than that, then it essentially truncates the file.
You could be missing lots of links that appear later in
the page.
max.outlinks.per.page -- by default it's 100. You might
want to increase this since for pages with something like
a nested navigation sidebar with tons of links, it won't
get any links from the main part of the page.
The *.xml files are fairly descriptive. So just reading through
them can be pretty helpful. I don't know if there is a full
guide to the config files.
Howie
>
>1)
>I did several testing running to fetch page from two
>website. The fetching depth is 10.
>
>After checking log files, I found the actual fetched
>page linkage is very different for two sites.
>
>In one site with lots of news, only first two depth
>fetching running well and only fetching 5 linkages.
>The actual linkages in that site is far beyond that.
>
>The other site can fetch till 10 rounds and fetched
>100's linkage.
>
>I wonder if any one has similar experience. Should I
>setup configure files in /conf/?
>
>2)
>Also, in Nutch/conf/ directory, I found several
>configuration files. Actually, I only modify
>crawl-urlfilter.txt to let it accept all the url
>(*.*).
>
>Is it proper?
>
>I really doesn't touch other conf files. Is there a
>guideline how I use these files?
>
>thanks,
>
>Michael,
>
>
>
>__________________________________________________
>Do You Yahoo!?
>Tired of spam? Yahoo! Mail has the best spam protection around
>http://mail.yahoo.com
Nutch's intranet VS internet crawling
Posted by "Feng (Michael) Ji" <fj...@yahoo.com>.
I wonder if there is any difference between these two?
Or intranet crawling must indicate an intranet site
explicitly in crawl-urlfilter.txt under /conf ?
thanks,
Michael,
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com