You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Rod Taylor <rb...@sitesell.com> on 2005/10/21 03:12:26 UTC
NDFS Limitations or Bug
We have been playing with a SegmentReader (we have a working copy) and
may have run into a limitation or bug within NDFS. Occurs on mapred code
for both the 18th and 20th.
When running the attached SegmentReader.java on a small segment (10k
urls or less) it runs problem free but on larger segments the NameNode
seems to go a little crazy (gets stuck in some kind of a loop).
The exact same code writing to stdout functions as expected even when a
segment is several million URLs or tens of GBs.
segread.op is the client output
Below is the NameNode output. These exceptions continue until you kill
the daemon.
java.io.IOException: Cannot abandon block during write
to /opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/.dump.crc
at
org.apache.nutch.ndfs.NameNode.abandonBlock(NameNode.java:149)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.nutch.ipc.RPC$1.call(RPC.java:186)
at org.apache.nutch.ipc.Server
$Handler.run(Server.java:198)
051019 174056 Server handler on 5466 call error:
java.io.IOException: Cannot abandon block during write
to /opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/.dump.crc
java.io.IOException: Cannot abandon block during write
to /opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/.dump.crc
at
org.apache.nutch.ndfs.NameNode.abandonBlock(NameNode.java:149)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.nutch.ipc.RPC$1.call(RPC.java:186)
at org.apache.nutch.ipc.Server
$Handler.run(Server.java:198)
051019 174056 Server handler on 5466 call error:
java.io.IOException: Cannot abandon block during write
to /opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/.dump.crc
java.io.IOException: Cannot abandon block during write
to /opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/.dump.crc
at
org.apache.nutch.ndfs.NameNode.abandonBlock(NameNode.java:149)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.nutch.ipc.RPC$1.call(RPC.java:186)
at org.apache.nutch.ipc.Server
$Handler.run(Server.java:198)
--
Rod Taylor <rb...@sitesell.com>