You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Rod Taylor <rb...@sitesell.com> on 2005/10/21 03:12:26 UTC

NDFS Limitations or Bug

We have been playing with a SegmentReader (we have a working copy) and
may have run into a limitation or bug within NDFS. Occurs on mapred code
for both the 18th and 20th.

When running the attached SegmentReader.java on a small segment (10k
urls or less) it runs problem free but on larger segments the NameNode
seems to go a little crazy (gets stuck in some kind of a loop).

The exact same code writing to stdout functions as expected even when a
segment is several million URLs or tens of GBs.

segread.op is the client output

Below is the NameNode output. These exceptions continue until you kill
the daemon.
        java.io.IOException: Cannot abandon block during write
        to /opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/.dump.crc
                at
        org.apache.nutch.ndfs.NameNode.abandonBlock(NameNode.java:149)
                at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
        Source)
                at
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:585)
                at org.apache.nutch.ipc.RPC$1.call(RPC.java:186)
                at org.apache.nutch.ipc.Server
        $Handler.run(Server.java:198)
        051019 174056 Server handler on 5466 call error:
        java.io.IOException: Cannot abandon block during write
        to /opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/.dump.crc
        java.io.IOException: Cannot abandon block during write
        to /opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/.dump.crc
                at
        org.apache.nutch.ndfs.NameNode.abandonBlock(NameNode.java:149)
                at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
        Source)
                at
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:585)
                at org.apache.nutch.ipc.RPC$1.call(RPC.java:186)
                at org.apache.nutch.ipc.Server
        $Handler.run(Server.java:198)
        051019 174056 Server handler on 5466 call error:
        java.io.IOException: Cannot abandon block during write
        to /opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/.dump.crc
        java.io.IOException: Cannot abandon block during write
        to /opt/sitesell/sbider_data/nutch/segments/20051019042119/20051019042121/segdump/.dump.crc
                at
        org.apache.nutch.ndfs.NameNode.abandonBlock(NameNode.java:149)
                at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
        Source)
                at
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:585)
                at org.apache.nutch.ipc.RPC$1.call(RPC.java:186)
                at org.apache.nutch.ipc.Server
        $Handler.run(Server.java:198)

-- 
Rod Taylor <rb...@sitesell.com>