You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by glumet <ja...@gmail.com> on 2014/03/15 22:26:27 UTC
Nutch 2.1 - fetching is not working (maybe broken generate?)
Hello everybody. I have one problem with my current Nutch 2.1 configuration.
I run my crawling process (throught bin/crawl script) every morning during
the last few months and everything went fine - I mean that in every run the
script generated new batch which was fetched and after that parsed and
finally indexed into Solr (I use also hBase as my data store).
I do not know why but something had broken and nothing is fetched now (also
because this problem there is nothing to parse and index).
If it could be helpful, I tried to do bin/nutch readdb -stats and output is
here: http://pastebin.com/r3vFK4T4 <http://pastebin.com/r3vFK4T4>
There are some things like
status 3 (status_gone): 9907
status 1 (status_unfetched): 6798166
status 0 (null): 2
but I dont understand too much this attributes... but 6798166 unfetched urls
seem very suspicious.
During crawling process I can see that GeneratorJob generates some batch id
but I dont know if this batchId is not empty or something.
/GeneratorJob: generated batch id: 1329930901-1268252438
FetcherJob: starting
FetcherJob : timelimit set for : -1
FetcherJob: threads: 10
FetcherJob: parsing: false
FetcherJob: resuming: false
FetcherJob: batchId: 1329930901-1268252438
Using queue mode : byHost
.
.
.
-activeThreads=0, spinWaiting=0, fetchQueues= 0, fetchQueues.totalSize=0
-activeThreads=0
FetcherJob: done /
fetching is really fast because as I said there is no url to be fetched.
Can you help me or give any advice?
Thanks, Jan
--
View this message in context: http://lucene.472066.n3.nabble.com/Nutch-2-1-fetching-is-not-working-maybe-broken-generate-tp4123813.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Nutch 2.1 - fetching is not working (maybe broken generate?)
Posted by glumet <ja...@gmail.com>.
To finish this thread, I solved this problem with upgrading to Nutch 2.x from
svn.
--
View this message in context: http://lucene.472066.n3.nabble.com/Nutch-2-1-fetching-is-not-working-maybe-broken-generate-tp4123813p4135152.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Nutch 2.1 - fetching is not working (maybe broken generate?)
Posted by glumet <ja...@gmail.com>.
Here it is thanks http://pastebin.com/BA2zgteJ
<http://pastebin.com/BA2zgteJ>
--
View this message in context: http://lucene.472066.n3.nabble.com/Nutch-2-1-fetching-is-not-working-maybe-broken-generate-tp4123813p4129520.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Nutch 2.1 - fetching is not working (maybe broken generate?)
Posted by Talat Uyarer <ta...@uyarer.com>.
I cant get your attachments Jan. Could you share on pastebin etc.
31 Mar 2014 19:14 tarihinde "glumet" <ja...@gmail.com> yazdı:
> My nutch-site.xml is here and nutch-default.xml here nutch-default.xml
> <http://lucene.472066.n3.nabble.com/file/n4128217/nutch-default.xml> .
>
> But there is fetcher limit deactivated in nutch-default.
>
>
> Thanks
>
>
> (nutch site if the link wouldn't work correctly,nutch default shouldn be
> changed)
>
>
>
> Many thanks
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Nutch-2-1-fetching-is-not-working-maybe-broken-generate-tp4123813p4128217.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
Re: Nutch 2.1 - fetching is not working (maybe broken generate?)
Posted by glumet <ja...@gmail.com>.
My nutch-site.xml is here and nutch-default.xml here nutch-default.xml
<http://lucene.472066.n3.nabble.com/file/n4128217/nutch-default.xml> .
But there is fetcher limit deactivated in nutch-default.
Thanks
(nutch site if the link wouldn't work correctly,nutch default shouldn be
changed)
Many thanks
--
View this message in context: http://lucene.472066.n3.nabble.com/Nutch-2-1-fetching-is-not-working-maybe-broken-generate-tp4123813p4128217.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Nutch 2.1 - fetching is not working (maybe broken generate?)
Posted by Talat Uyarer <ta...@uyarer.com>.
You set a time limit for generate. These urls are hitted by time limit. Can
you share your nutch-site.xml
If you change nutch-defualt.xml please change with orginal version before.
24 Mar 2014 20:25 tarihinde "glumet" <ja...@gmail.com> yazdı:
> Hi,
> I generated batchid 1111 with topN 100, in Hbase I have 100 rows with
> generate mark with the same value, but fetch is not working, after run
> nutch* fetch 1111* I got no progres for this generated urls, in database
> is
> no row with fetch mark and pages are unfetched.
> I write debug prints in nutch source codes and build it. FetcherMapper
> works
> fine, if generate mark is 1111 writes it in cotext (/context.write(new
> IntWritable(random.nextInt(65536)),
> new
> FetchEntry(context.getConfiguration(), key, page));)/, but in
> FetcherReducer is nothing /*fit = fetchQueues.getFetchItem(); */ produces
> null /(org.apache.nutch.fetcher.FetcherReducer$FetcherThread getFetchItem :
> null/
> )
> It looks like the queues are empty, but why? What could be problem here?
> Thanks in advance
>
> /FetcherJob: starting
> FetcherJob: batchId: 1111
> FetcherJob: threads: 10
> FetcherJob: parsing: false
> FetcherJob: resuming: false
> FetcherJob : timelimit set for : 1395683736716
> Using queue mode : byHost
> Fetcher: threads: 10
> QueueFeeder finished: total 0 records. Hit by time limit :100
> -finishing thread FetcherThread0, activeThreads=0
> -finishing thread FetcherThread2, activeThreads=1
> -finishing thread FetcherThread1, activeThreads=0
> -finishing thread FetcherThread4, activeThreads=0
> -finishing thread FetcherThread3, activeThreads=0
> -finishing thread FetcherThread5, activeThreads=0
> -finishing thread FetcherThread6, activeThreads=0
> -finishing thread FetcherThread7, activeThreads=0
> -finishing thread FetcherThread8, activeThreads=0
> Fetcher: throughput threshold: -1
> -finishing thread FetcherThread9, activeThreads=0
> Fetcher: throughput threshold sequence: 5
> 0/0 spinwaiting/active, 0 pages, 0 errors, 0.0 0 pages/s, 0 0 kb/s, 0 URLs
> in 0 queues
> -activeThreads=0
> FetcherJob: done/
>
> HBase shell: return 100 rows only for generate mark
> scan 'webpage_webpage', {COLUMNS => ['mk:_gnmrk_','mk:_ftcmrk_']}
>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Nutch-2-1-fetching-is-not-working-maybe-broken-generate-tp4123813p4126652.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
Re: Nutch 2.1 - fetching is not working (maybe broken generate?)
Posted by glumet <ja...@gmail.com>.
Hi,
I generated batchid 1111 with topN 100, in Hbase I have 100 rows with
generate mark with the same value, but fetch is not working, after run
nutch* fetch 1111* I got no progres for this generated urls, in database is
no row with fetch mark and pages are unfetched.
I write debug prints in nutch source codes and build it. FetcherMapper works
fine, if generate mark is 1111 writes it in cotext (/context.write(new
IntWritable(random.nextInt(65536)),
new FetchEntry(context.getConfiguration(), key, page));)/, but in
FetcherReducer is nothing /*fit = fetchQueues.getFetchItem(); */ produces
null /(org.apache.nutch.fetcher.FetcherReducer$FetcherThread getFetchItem :
null/
)
It looks like the queues are empty, but why? What could be problem here?
Thanks in advance
/FetcherJob: starting
FetcherJob: batchId: 1111
FetcherJob: threads: 10
FetcherJob: parsing: false
FetcherJob: resuming: false
FetcherJob : timelimit set for : 1395683736716
Using queue mode : byHost
Fetcher: threads: 10
QueueFeeder finished: total 0 records. Hit by time limit :100
-finishing thread FetcherThread0, activeThreads=0
-finishing thread FetcherThread2, activeThreads=1
-finishing thread FetcherThread1, activeThreads=0
-finishing thread FetcherThread4, activeThreads=0
-finishing thread FetcherThread3, activeThreads=0
-finishing thread FetcherThread5, activeThreads=0
-finishing thread FetcherThread6, activeThreads=0
-finishing thread FetcherThread7, activeThreads=0
-finishing thread FetcherThread8, activeThreads=0
Fetcher: throughput threshold: -1
-finishing thread FetcherThread9, activeThreads=0
Fetcher: throughput threshold sequence: 5
0/0 spinwaiting/active, 0 pages, 0 errors, 0.0 0 pages/s, 0 0 kb/s, 0 URLs
in 0 queues
-activeThreads=0
FetcherJob: done/
HBase shell: return 100 rows only for generate mark
scan 'webpage_webpage', {COLUMNS => ['mk:_gnmrk_','mk:_ftcmrk_']}
--
View this message in context: http://lucene.472066.n3.nabble.com/Nutch-2-1-fetching-is-not-working-maybe-broken-generate-tp4123813p4126652.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Nutch 2.1 - fetching is not working (maybe broken generate?)
Posted by anupamk <an...@usc.edu>.
I m sorry for causing confusion.
Nutch 2.x fetch cycle is significantly different in terms of implementation
than Nutch 1.x
The idea is pretty much the same though.
So, the idea to check whether the each step works is as follows:
1. use generatorjob to generate a batch job that will be fetch topN = 100
links from your webtable
2. check your webtable and see which links have a generator mark -- if you
have generator mark on 100 links then your generate job is work
3. use fetcherjob to fetch -- once the job is completed see which links in
the webtable has fetched mark
4. use parserjob to parse -- and then check webtable is updated correctly or
not after the job
5. use updateDB job to update webtable -- and then check webtable is updated
correctly or not
you get the idea right ?
In nutch 1.x we don't use gora or have webtable and every operation relies
on file operations -- that's the reason you readseg and create segments etc.
The whole segment thing is no longer there in nutch2.x because you have
webtable in gora taking its place.
However, since I have only worked on Nutch1.x (and still do) I can't help
you on how to check webpage table for generator/fetcher/parser markers etc
...
But I am sure you will be able to documentation in the wiki on the commands
how to do it.
here are a few good place to get started
http://wiki.apache.org/nutch/Nutch2Crawling
http://nlp.solutions.asia/?p=232
--
View this message in context: http://lucene.472066.n3.nabble.com/Nutch-2-1-fetching-is-not-working-maybe-broken-generate-tp4123813p4124590.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Nutch 2.1 - fetching is not working (maybe broken generate?)
Posted by glumet <ja...@gmail.com>.
Ok, now I can see that "readseg" is not supported in Nutch 2.1. I can
manipulate only with these commands:
xbouj19@ir:/opt/ir/nutch$ sudo ./bin/nutch
Usage: nutch COMMAND
where COMMAND is one of:
inject inject new urls into the database
hostinject creates or updates an existing host table from a text file
generate generate new batches to fetch from crawl db
fetch fetch URLs marked during generate
parse parse URLs marked during fetch
updatedb update web table after parsing
updatehostdb update host table after parsing
readdb read/dump records from page database
readhostdb display entries from the hostDB
elasticindex run the elasticsearch indexer
solrindex run the solr indexer on parsed batches
solrdedup remove duplicates from solr
parsechecker check the parser for a given url
indexchecker check the indexing filters for a given url
plugin load a plugin and run one of its classes main()
nutchserver run a (local) Nutch server on a user defined port
junit runs the given JUnit test
or
CLASSNAME run the class named CLASSNAME
Most commands print help when invoked w/o parameters.
--
View this message in context: http://lucene.472066.n3.nabble.com/Nutch-2-1-fetching-is-not-working-maybe-broken-generate-tp4123813p4124523.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Nutch 2.1 - fetching is not working (maybe broken generate?)
Posted by glumet <ja...@gmail.com>.
Hi, thank you for your reply. When I look into hadoop.log, I can see
2014-02-22 16:16:19,174 INFO mapreduce.GoraRecordReader -
gora.buffer.read.limit = 10000
2014-02-22 16:16:21,284 INFO mapreduce.GoraRecordReader -
gora.buffer.read.limit = 10000
etc. etc. ... this repeats many times... hbase.log also seems fine.
I will try your "step by step" advices and write here later.
About filters... only filter that I use is regex-urlfilter.txt and I did not
change it.
--
View this message in context: http://lucene.472066.n3.nabble.com/Nutch-2-1-fetching-is-not-working-maybe-broken-generate-tp4123813p4124514.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Nutch 2.1 - fetching is not working (maybe broken generate?)
Posted by anupamk <an...@usc.edu>.
have you seen the output of hadoop.log ?
Any exceptions that might have occured during
generate / fetch / parse phases would be logged there.
--
View this message in context: http://lucene.472066.n3.nabble.com/Nutch-2-1-fetching-is-not-working-maybe-broken-generate-tp4123813p4123830.html
Sent from the Nutch - User mailing list archive at Nabble.com.