You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Sunnyvale Fl <su...@gmail.com> on 2006/02/16 00:21:04 UTC

impossible situation error

Can someone clue me in here please?  I really cannot decode what is going
on.  I keep getting this impossible situation error that says there is a
"score-edit" for a page that comes after "the current page".  I am running
nutch 0.7.1, and starting with a fresh db and fresh segment directory.  Here
is an example:

starting url: http://www.legis.state.ak.us/default.htm
prefix-urlfilter: http://www.legis.state.ak.us/

error message:
java.io.IOException: Impossible situation.  There is a score-edit for
http://www.legi
s.state.ak.us/fhelp/helpmain.htm, which comes after the current Page
http://www.legis.state.ak.us/folhome.htm

Re: impossible situation error

Posted by Sunnyvale Fl <su...@gmail.com>.

logging at info gives a rather long stack...  here it is.
the funny thing is, after the crash, i deleted the tmp-dir in the db folder
(without cleaning up the segment folders), and recrawl the same site, and it
worked without crashing.

060215 151338 Using URL normalizer: org.apache.nutch.net.BasicUrlNormalizer
060215 151338 Added 1 pages
060215 151338 Processing pagesByURL: Sorted 1 instructions in 0.01 seconds.
060215 151338 Processing pagesByURL: Sorted 100.0 instructions/second
060215 151338 Processing pagesByURL: Merged to new DB containing 1 records
in 0.01 seconds
060215 151338 Processing pagesByURL: Merged 100.0 records/second
060215 151338 Processing pagesByMD5: Sorted 1 instructions in 0.0 seconds.
060215 151338 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151338 Processing pagesByMD5: Merged to new DB containing 1 records
in 0.0 seconds
060215 151338 Processing pagesByMD5: Merged Infinity records/second
060215 151338 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs.
060215 151338 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs.
060215 151338 Processing pagesByURL: Sorted 1 instructions in 0.0 seconds.
060215 151338 Processing pagesByURL: Sorted Infinity instructions/second
060215 151338 Processing pagesByURL: Merged to new DB containing 1 records
in 0.0 seconds
060215 151338 Processing pagesByURL: Merged Infinity records/second
060215 151338 Processing pagesByMD5: Sorted 1 instructions in 0.0 seconds.
060215 151338 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151338 Processing pagesByMD5: Merged to new DB containing 1 records
in 0.0 seconds
060215 151338 Processing pagesByMD5: Merged Infinity records/second
060215 151338 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs.
060215 151338 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs.
060215 151338 Processing
/var/nutch/indexes-ak/segments/20060215151338/fetchlist.unsorted: Sorted 1
entries in 0.0 seconds.
060215 151338 Processing
/var/nutch/indexes-ak/segments/20060215151338/fetchlist.unsorted: Sorted
Infinity entries/second
060215 151338 Overall processing: Sorted 1 entries in 0.0 seconds.
060215 151338 Overall processing: Sorted 0.0 entries/second
060215 151339 logging at INFO
060215 151339 fetching http://www.legis.state.ak.us/default.htm
060215 151339 http.proxy.host = null
060215 151339 http.proxy.port = 8080
060215 151339 http.timeout = 10000
060215 151339 http.content.limit = -1
060215 151339 http.agent = NutchCVS/0.7.1 (Nutch;
http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
060215 151339 fetcher.server.delay = 1000
060215 151339 http.max.delays = 25
060215 151340 status: segment 20060215151338, 1 pages, 0 errors, 8287 bytes,
1020 ms
060215 151340 status: 0.98039216 pages/s, 63.472733 kb/s, 8287.0 bytes/page
060215 151341 Updating /var/nutch/indexes-ak/db
060215 151341 Updating for /var/nutch/indexes-ak/segments/20060215151338
060215 151341 Processing document 0
060215 151341 Finishing update
060215 151341 Processing pagesByURL: Sorted 6 instructions in 0.0 seconds.
060215 151341 Processing pagesByURL: Sorted Infinity instructions/second
060215 151341 Processing pagesByURL: Merged to new DB containing 6 records
in 0.0 seconds
060215 151341 Processing pagesByURL: Merged Infinity records/second
060215 151341 Processing pagesByMD5: Sorted 7 instructions in 0.0 seconds.
060215 151341 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151341 Processing pagesByMD5: Merged to new DB containing 6 records
in 0.0 seconds
060215 151341 Processing pagesByMD5: Merged Infinity records/second
060215 151341 Processing linksByMD5: Sorted 6 instructions in 0.0 seconds.
060215 151341 Processing linksByMD5: Sorted Infinity instructions/second
060215 151341 Processing linksByMD5: Merged to new DB containing 5 records
in 0.01 seconds
060215 151341 Processing linksByMD5: Merged 500.0 records/second
060215 151341 Processing linksByURL: Sorted 5 instructions in 0.0 seconds.
060215 151341 Processing linksByURL: Sorted Infinity instructions/second
060215 151341 Processing linksByURL: Merged to new DB containing 5 records
in 0.0 seconds
060215 151341 Processing linksByURL: Merged Infinity records/second
060215 151341 Processing linksByMD5: Sorted 5 instructions in 0.0 seconds.
060215 151341 Processing linksByMD5: Sorted Infinity instructions/second
060215 151341 Processing linksByMD5: Merged to new DB containing 5 records
in 0.0 seconds
060215 151341 Processing linksByMD5: Merged Infinity records/second
060215 151341 Update finished
060215 151341 Processing pagesByURL: Sorted 5 instructions in 0.0 seconds.
060215 151341 Processing pagesByURL: Sorted Infinity instructions/second
060215 151341 Processing pagesByURL: Merged to new DB containing 6 records
in 0.01 seconds
060215 151341 Processing pagesByURL: Merged 600.0 records/second
060215 151341 Processing pagesByMD5: Sorted 5 instructions in 0.0 seconds.
060215 151341 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151341 Processing pagesByMD5: Merged to new DB containing 6 records
in 0.0 seconds
060215 151341 Processing pagesByMD5: Merged Infinity records/second
060215 151341 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs.
060215 151341 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs.
060215 151341 Processing
/var/nutch/indexes-ak/segments/20060215151341/fetchlist.unsorted: Sorted 5
entries in 0.0 seconds.
060215 151341 Processing
/var/nutch/indexes-ak/segments/20060215151341/fetchlist.unsorted: Sorted
Infinity entries/second
060215 151341 Overall processing: Sorted 5 entries in 0.0 seconds.
060215 151341 Overall processing: Sorted 0.0 entries/second
060215 151341 logging at INFO
060215 151341 fetching http://www.legis.state.ak.us/infolist.htm
060215 151341 fetching http://www.legis.state.ak.us/home.htm
060215 151341 fetching http://www.legis.state.ak.us/fhelp/helpmain.htm
060215 151341 fetching http://www.legis.state.ak.us/FBframe.htm
060215 151341 fetching http://www.legis.state.ak.us/Legframe.htm
060215 151341 fetch of
http://www.legis.state.ak.us/fhelp/helpmain.htmfailed with:
java.lang.Exception: org.apache.nutch.protocol.R
esourceGone: Blocked by robots.txt
060215 151342 status: segment 20060215151341, 4 pages, 1 errors, 16047
bytes, 1010 ms
060215 151342 status: 3.960396 pages/s, 124.12593 kb/s, 4011.75 bytes/page
060215 151343 Updating /var/nutch/indexes-ak/db
060215 151343 Updating for /var/nutch/indexes-ak/segments/20060215151341
060215 151343 Processing document 0
060215 151343 Finishing update
060215 151343 Processing pagesByURL: Sorted 10 instructions in 0.0 seconds.
060215 151343 Processing pagesByURL: Sorted Infinity instructions/second
060215 151343 Processing pagesByURL: Merged to new DB containing 9 records
in 0.0 seconds
060215 151343 Processing pagesByURL: Merged Infinity records/second
060215 151343 Processing pagesByMD5: Sorted 13 instructions in 0.0 seconds.
060215 151343 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151343 Processing pagesByMD5: Merged to new DB containing 9 records
in 0.0 seconds
060215 151343 Processing pagesByMD5: Merged Infinity records/second
060215 151343 Processing linksByMD5: Sorted 11 instructions in 0.0 seconds.
060215 151343 Processing linksByMD5: Sorted Infinity instructions/second
060215 151343 Processing linksByMD5: Merged to new DB containing 11 records
in 0.0 seconds
060215 151343 Processing linksByMD5: Merged Infinity records/second
060215 151343 Processing linksByURL: Sorted 6 instructions in 0.0 seconds.
060215 151343 Processing linksByURL: Sorted Infinity instructions/second
060215 151343 Processing linksByURL: Merged to new DB containing 9 records
in 0.01 seconds
060215 151343 Processing linksByURL: Merged 900.0 records/second
060215 151343 Processing linksByMD5: Sorted 9 instructions in 0.0 seconds.
060215 151343 Processing linksByMD5: Sorted Infinity instructions/second
060215 151343 Processing linksByMD5: Merged to new DB containing 9 records
in 0.0 seconds
060215 151343 Processing linksByMD5: Merged Infinity records/second
060215 151343 Update finished
060215 151343 Processing pagesByURL: Sorted 4 instructions in 0.0 seconds.
060215 151343 Processing pagesByURL: Sorted Infinity instructions/second
060215 151343 Processing pagesByURL: Merged to new DB containing 9 records
in 0.0 seconds
060215 151343 Processing pagesByURL: Merged Infinity records/second
060215 151343 Processing pagesByMD5: Sorted 4 instructions in 0.0 seconds.
060215 151343 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151343 Processing pagesByMD5: Merged to new DB containing 9 records
in 0.0 seconds
060215 151343 Processing pagesByMD5: Merged Infinity records/second
060215 151343 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs.
060215 151343 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs.
060215 151343 Processing
/var/nutch/indexes-ak/segments/20060215151343/fetchlist.unsorted: Sorted 4
entries in 0.0 seconds.
060215 151343 Processing
/var/nutch/indexes-ak/segments/20060215151343/fetchlist.unsorted: Sorted
Infinity entries/second
060215 151343 Overall processing: Sorted 4 entries in 0.0 seconds.
060215 151343 Overall processing: Sorted 0.0 entries/second
060215 151343 logging at INFO
060215 151343 fetching http://www.legis.state.ak.us/
060215 151343 fetching http://www.legis.state.ak.us/folhome.htm
060215 151343 fetching http://www.legis.state.ak.us/poms/
060215 151343 fetch of http://www.legis.state.ak.us/poms/ failed with:
java.lang.Exception: org.apache.nutch.protocol.ResourceGone:
Blocked by robots.txt
060215 151343 fetching http://www.legis.state.ak.us/basis/start.asp
060215 151343 fetch okay, but can't parse
http://www.legis.state.ak.us/basis/start.asp, reason: failed(2,200):
sun.io.MalformedInput
Exception: Missing byte-order mark
060215 151344 status: segment 20060215151343, 3 pages, 1 errors, 23109
bytes, 1010 ms
060215 151344 status: 2.970297 pages/s, 178.75154 kb/s, 7703.0 bytes/page
060215 151345 Updating /var/nutch/indexes-ak/db
060215 151345 Updating for /var/nutch/indexes-ak/segments/20060215151343
060215 151345 Processing document 0
060215 151345 Finishing update
060215 151345 Processing pagesByURL: Sorted 5 instructions in 0.0 seconds.
060215 151345 Processing pagesByURL: Sorted Infinity instructions/second
060215 151345 Processing pagesByURL: Merged to new DB containing 9 records
in 0.0 seconds
060215 151345 Processing pagesByURL: Merged Infinity records/second
060215 151345 Processing pagesByMD5: Sorted 8 instructions in 0.0 seconds.
060215 151345 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151345 Processing pagesByMD5: Merged to new DB containing 9 records
in 0.0 seconds
060215 151345 Processing pagesByMD5: Merged Infinity records/second
060215 151345 Processing linksByMD5: Sorted 13 instructions in 0.0 seconds.
060215 151345 Processing linksByMD5: Sorted Infinity instructions/second
060215 151345 Processing linksByMD5: Merged to new DB containing 14 records
in 0.0 seconds
060215 151345 Processing linksByMD5: Merged Infinity records/second
060215 151345 Processing linksByURL: Sorted 9 instructions in 0.0 seconds.
060215 151345 Processing linksByURL: Sorted Infinity instructions/second
060215 151345 Processing linksByURL: Merged to new DB containing 14 records
in 0.01 seconds
060215 151345 Processing linksByURL: Merged 1400.0 records/second
060215 151345 Processing linksByMD5: Sorted 10 instructions in 0.0 seconds.
060215 151345 Processing linksByMD5: Sorted Infinity instructions/second
060215 151345 Processing linksByMD5: Merged to new DB containing 14 records
in 0.0 seconds
060215 151345 Processing linksByMD5: Merged Infinity records/second
060215 151345 Update finished
060215 151345 Processing pagesByURL: Sorted 1 instructions in 0.0 seconds.
060215 151345 Processing pagesByURL: Sorted Infinity instructions/second
060215 151345 Processing pagesByURL: Merged to new DB containing 9 records
in 0.0 seconds
060215 151345 Processing pagesByURL: Merged Infinity records/second
060215 151345 Processing pagesByMD5: Sorted 1 instructions in 0.0 seconds.
060215 151345 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151345 Processing pagesByMD5: Merged to new DB containing 9 records
in 0.0 seconds
060215 151345 Processing pagesByMD5: Merged Infinity records/second
060215 151345 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs.
060215 151345 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs.
060215 151345 Processing
/var/nutch/indexes-ak/segments/20060215151345/fetchlist.unsorted: Sorted 1
entries in 0.0 seconds.
060215 151345 Processing
/var/nutch/indexes-ak/segments/20060215151345/fetchlist.unsorted: Sorted
Infinity entries/second
060215 151345 Overall processing: Sorted 1 entries in 0.0 seconds.
060215 151345 Overall processing: Sorted 0.0 entries/second
060215 151345 logging at INFO
060215 151345 fetching http://www.legis.state.ak.us/fhelp/helpmain.htm
060215 151345 fetch of
http://www.legis.state.ak.us/fhelp/helpmain.htmfailed with:
java.lang.Exception: org.apache.nutch.protocol.R
esourceGone: Blocked by robots.txt
060215 151346 Updating /var/nutch/indexes-ak/db
060215 151346 Updating for /var/nutch/indexes-ak/segments/20060215151345
060215 151346 Processing document 0
060215 151346 Finishing update
060215 151346 Processing pagesByURL: Sorted 1 instructions in 0.0 seconds.
060215 151346 Processing pagesByURL: Sorted Infinity instructions/second
060215 151346 Processing pagesByURL: Merged to new DB containing 8 records
in 0.0 seconds
060215 151346 Processing pagesByURL: Merged Infinity records/second
060215 151346 Processing pagesByMD5: Sorted 1 instructions in 0.0 seconds.
060215 151346 Processing pagesByMD5: Sorted Infinity instructions/second
060215 151346 Processing pagesByMD5: Merged to new DB containing 8 records
in 0.0 seconds
060215 151346 Processing pagesByMD5: Merged Infinity records/second
060215 151346 Processing linksByMD5: Sorted 1 instructions in 0.01 seconds.
060215 151346 Processing linksByMD5: Sorted 100.0 instructions/second
060215 151346 Processing linksByMD5: Merged to new DB containing 14 records
in 0.0 seconds
060215 151346 Processing linksByMD5: Merged Infinity records/second
060215 151346 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs.
060215 151346 Update finished
060215 151346 Overall processing: Sorted 0 entries in 0.0 seconds.
060215 151346 Overall processing: Sorted NaN entries/second
060215 151346 Deleted empty fetchlist segment dir
/var/nutch/indexes-ak/segments/20060215151346
060215 151346 Start at: 0
060215 151346 Extent: 8
060215 151346 Pages consumed: 0 (at index 0). Links fetched: 0.
060215 151346 Opened stream to file 0
060215 151346 Pages written: 0
060215 151346 Processing pagesByURL: Sorted 5 instructions in 0.0 seconds.
060215 151346 Processing pagesByURL: Sorted Infinity instructions/second
060215 151346 Processing pagesByURL: Merged to new DB containing 8 records
in 0.0 seconds
060215 151346 Processing pagesByURL: Merged Infinity records/second
060215 151346 Processing pagesByMD5: Sorted 5 instructions in 0.01 seconds.
060215 151346 Processing pagesByMD5: Sorted 500.0 instructions/second
060215 151346 Processing pagesByMD5: Merged to new DB containing 8 records
in 0.0 seconds
060215 151346 Processing pagesByMD5: Merged Infinity records/second
060215 151346 Processing linksByMD5: Copied file (4096 bytes) in 0.0 secs.
060215 151346 Processing linksByURL: Copied file (4096 bytes) in 0.0 secs.
060215 151346 java.io.IOException: Impossible situation.  There is a
score-edit for http://www.legis.state.ak.us/fhelp/helpmain.htm,
 which comes after the current Page http://www.legis.state.ak.us/folhome.htm
060215 151346 at
org.apache.nutch.tools.DistributedAnalysisTool.completeRound(
DistributedAnalysisTool.java:532)
060215 151346 at org.apache.nutch.tools.LinkAnalysisTool.iterate(
LinkAnalysisTool.java:60)
060215 151346 at org.apache.nutch.tools.LinkAnalysisTool.main(
LinkAnalysisTool.java:81)

Re: impossible situation error

Posted by Sunnyvale Fl <su...@gmail.com>.

I am actually running version 0.7.1 - I thought the nightly builds are dev
version of 0.8?  Thanks...

On 2/16/06, Stefan Groschupf <sg...@media-style.com> wrote:
>
> Can you provide a full stack, may by setting the loglevel higher.
> I guess this is a problem of the new OPIC score calculation but you
> are the first that report such a problem.
> In general - sorry to repeat that often - it is a good idea to run
> the latest nightly builds of a open source application - at least
> that is my point of view.
>
> Am 16.02.2006 um 00:21 schrieb Sunnyvale Fl:
>
> > Can someone clue me in here please?  I really cannot decode what is
> > going
> > on.  I keep getting this impossible situation error that says there
> > is a
> > "score-edit" for a page that comes after "the current page".  I am
> > running
> > nutch 0.7.1, and starting with a fresh db and fresh segment
> > directory.  Here
> > is an example:
> >
> > starting url: http://www.legis.state.ak.us/default.htm
> > prefix-urlfilter: http://www.legis.state.ak.us/
> >
> > error message:
> > java.io.IOException: Impossible situation.  There is a score-edit for
> > http://www.legi
> > s.state.ak.us/fhelp/helpmain.htm, which comes after the current Page
> > http://www.legis.state.ak.us/folhome.htm
>
>

Re: impossible situation error

Posted by Stefan Groschupf <sg...@media-style.com>.

Can you provide a full stack, may by setting the loglevel higher.
I guess this is a problem of the new OPIC score calculation but you  
are the first that report such a problem.
In general - sorry to repeat that often - it is a good idea to run  
the latest nightly builds of a open source application - at least  
that is my point of view.

Am 16.02.2006 um 00:21 schrieb Sunnyvale Fl:

> Can someone clue me in here please?  I really cannot decode what is  
> going
> on.  I keep getting this impossible situation error that says there  
> is a
> "score-edit" for a page that comes after "the current page".  I am  
> running
> nutch 0.7.1, and starting with a fresh db and fresh segment  
> directory.  Here
> is an example:
>
> starting url: http://www.legis.state.ak.us/default.htm
> prefix-urlfilter: http://www.legis.state.ak.us/
>
> error message:
> java.io.IOException: Impossible situation.  There is a score-edit for
> http://www.legi
> s.state.ak.us/fhelp/helpmain.htm, which comes after the current Page
> http://www.legis.state.ak.us/folhome.htm