You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Nigel Daley (JIRA)" <ji...@apache.org> on 2007/06/22 03:46:25 UTC

[jira] Created: (HADOOP-1520) IndexOutOfBoundsException in FSEditLog.processIOError

IndexOutOfBoundsException in FSEditLog.processIOError
-----------------------------------------------------

                 Key: HADOOP-1520
                 URL: https://issues.apache.org/jira/browse/HADOOP-1520
             Project: Hadoop
          Issue Type: Bug
    Affects Versions: 0.14.0
            Reporter: Nigel Daley
            Assignee: Konstantin Shvachko


Running NNBench I saw these exceptions in the NameNode logs:

NAMENODE:
2007-06-21 04:02:53,587 INFO org.apache.hadoop.fs.FSNamesystem: Roll FSImage
2007-06-21 04:02:53,594 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
        at java.util.ArrayList.RangeCheck(ArrayList.java:547)
        at java.util.ArrayList.remove(ArrayList.java:387)
        at org.apache.hadoop.dfs.FSEditLog.processIOError(FSEditLog.java:169)
        at org.apache.hadoop.dfs.FSEditLog.logSync(FSEditLog.java:407)
        at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:878)
        at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:346)
        at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)

...

2007-06-21 04:02:53,823 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.completeFile: failed to complete /user/hadoopqa/nameNode100Benchmark/output/.214.crc because dir.getFileBlocks() is non-null and pendingFile is null
2007-06-21 04:02:53,828 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
        at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:352)
        at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)

This last exception occurs over and over again forever (this could be a result of the way NNBench is written).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-1520) IndexOutOfBoundsException in FSEditLog.processIOError

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-1520:
-------------------------------------

    Status: Patch Available  (was: Open)

Appropriate locking for the methods that the secondary namenode uses to roll the edit log and fsimage.

> IndexOutOfBoundsException in FSEditLog.processIOError
> -----------------------------------------------------
>
>                 Key: HADOOP-1520
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1520
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.14.0
>            Reporter: Nigel Daley
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: 1520.patch
>
>
> Running NNBench I saw these exceptions in the NameNode logs:
> NAMENODE:
> 2007-06-21 04:02:53,587 INFO org.apache.hadoop.fs.FSNamesystem: Roll FSImage
> 2007-06-21 04:02:53,594 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>         at java.util.ArrayList.remove(ArrayList.java:387)
>         at org.apache.hadoop.dfs.FSEditLog.processIOError(FSEditLog.java:169)
>         at org.apache.hadoop.dfs.FSEditLog.logSync(FSEditLog.java:407)
>         at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:878)
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:346)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> ...
> 2007-06-21 04:02:53,823 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.completeFile: failed to complete /user/hadoopqa/nameNode100Benchmark/output/.214.crc because dir.getFileBlocks() is non-null and pendingFile is null
> 2007-06-21 04:02:53,828 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
> java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:352)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> This last exception occurs over and over again forever (this could be a result of the way NNBench is written).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (HADOOP-1520) IndexOutOfBoundsException in FSEditLog.processIOError

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur reassigned HADOOP-1520:
----------------------------------------

    Assignee: dhruba borthakur  (was: Konstantin Shvachko)

> IndexOutOfBoundsException in FSEditLog.processIOError
> -----------------------------------------------------
>
>                 Key: HADOOP-1520
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1520
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.14.0
>            Reporter: Nigel Daley
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: 1520.patch
>
>
> Running NNBench I saw these exceptions in the NameNode logs:
> NAMENODE:
> 2007-06-21 04:02:53,587 INFO org.apache.hadoop.fs.FSNamesystem: Roll FSImage
> 2007-06-21 04:02:53,594 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>         at java.util.ArrayList.remove(ArrayList.java:387)
>         at org.apache.hadoop.dfs.FSEditLog.processIOError(FSEditLog.java:169)
>         at org.apache.hadoop.dfs.FSEditLog.logSync(FSEditLog.java:407)
>         at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:878)
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:346)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> ...
> 2007-06-21 04:02:53,823 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.completeFile: failed to complete /user/hadoopqa/nameNode100Benchmark/output/.214.crc because dir.getFileBlocks() is non-null and pendingFile is null
> 2007-06-21 04:02:53,828 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
> java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:352)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> This last exception occurs over and over again forever (this could be a result of the way NNBench is written).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-1520) IndexOutOfBoundsException in FSEditLog.processIOError

Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12508977 ] 

Hairong Kuang commented on HADOOP-1520:
---------------------------------------

+1 The patch looks good.

> IndexOutOfBoundsException in FSEditLog.processIOError
> -----------------------------------------------------
>
>                 Key: HADOOP-1520
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1520
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.14.0
>            Reporter: Nigel Daley
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: 1520.patch
>
>
> Running NNBench I saw these exceptions in the NameNode logs:
> NAMENODE:
> 2007-06-21 04:02:53,587 INFO org.apache.hadoop.fs.FSNamesystem: Roll FSImage
> 2007-06-21 04:02:53,594 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>         at java.util.ArrayList.remove(ArrayList.java:387)
>         at org.apache.hadoop.dfs.FSEditLog.processIOError(FSEditLog.java:169)
>         at org.apache.hadoop.dfs.FSEditLog.logSync(FSEditLog.java:407)
>         at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:878)
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:346)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> ...
> 2007-06-21 04:02:53,823 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.completeFile: failed to complete /user/hadoopqa/nameNode100Benchmark/output/.214.crc because dir.getFileBlocks() is non-null and pendingFile is null
> 2007-06-21 04:02:53,828 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
> java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:352)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> This last exception occurs over and over again forever (this could be a result of the way NNBench is written).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-1520) IndexOutOfBoundsException in FSEditLog.processIOError

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-1520:
-------------------------------------

    Component/s: dfs

> IndexOutOfBoundsException in FSEditLog.processIOError
> -----------------------------------------------------
>
>                 Key: HADOOP-1520
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1520
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.14.0
>            Reporter: Nigel Daley
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.14.0
>
>
> Running NNBench I saw these exceptions in the NameNode logs:
> NAMENODE:
> 2007-06-21 04:02:53,587 INFO org.apache.hadoop.fs.FSNamesystem: Roll FSImage
> 2007-06-21 04:02:53,594 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>         at java.util.ArrayList.remove(ArrayList.java:387)
>         at org.apache.hadoop.dfs.FSEditLog.processIOError(FSEditLog.java:169)
>         at org.apache.hadoop.dfs.FSEditLog.logSync(FSEditLog.java:407)
>         at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:878)
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:346)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> ...
> 2007-06-21 04:02:53,823 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.completeFile: failed to complete /user/hadoopqa/nameNode100Benchmark/output/.214.crc because dir.getFileBlocks() is non-null and pendingFile is null
> 2007-06-21 04:02:53,828 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
> java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:352)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> This last exception occurs over and over again forever (this could be a result of the way NNBench is written).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-1520) IndexOutOfBoundsException in FSEditLog.processIOError

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12509285 ] 

Hudson commented on HADOOP-1520:
--------------------------------

Integrated in Hadoop-Nightly #140 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/140/])

> IndexOutOfBoundsException in FSEditLog.processIOError
> -----------------------------------------------------
>
>                 Key: HADOOP-1520
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1520
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.14.0
>            Reporter: Nigel Daley
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: 1520.patch
>
>
> Running NNBench I saw these exceptions in the NameNode logs:
> NAMENODE:
> 2007-06-21 04:02:53,587 INFO org.apache.hadoop.fs.FSNamesystem: Roll FSImage
> 2007-06-21 04:02:53,594 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>         at java.util.ArrayList.remove(ArrayList.java:387)
>         at org.apache.hadoop.dfs.FSEditLog.processIOError(FSEditLog.java:169)
>         at org.apache.hadoop.dfs.FSEditLog.logSync(FSEditLog.java:407)
>         at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:878)
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:346)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> ...
> 2007-06-21 04:02:53,823 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.completeFile: failed to complete /user/hadoopqa/nameNode100Benchmark/output/.214.crc because dir.getFileBlocks() is non-null and pendingFile is null
> 2007-06-21 04:02:53,828 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
> java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:352)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> This last exception occurs over and over again forever (this could be a result of the way NNBench is written).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-1520) IndexOutOfBoundsException in FSEditLog.processIOError

Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-1520:
--------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I just commited this. Thanks Dhruba!

> IndexOutOfBoundsException in FSEditLog.processIOError
> -----------------------------------------------------
>
>                 Key: HADOOP-1520
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1520
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.14.0
>            Reporter: Nigel Daley
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: 1520.patch
>
>
> Running NNBench I saw these exceptions in the NameNode logs:
> NAMENODE:
> 2007-06-21 04:02:53,587 INFO org.apache.hadoop.fs.FSNamesystem: Roll FSImage
> 2007-06-21 04:02:53,594 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>         at java.util.ArrayList.remove(ArrayList.java:387)
>         at org.apache.hadoop.dfs.FSEditLog.processIOError(FSEditLog.java:169)
>         at org.apache.hadoop.dfs.FSEditLog.logSync(FSEditLog.java:407)
>         at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:878)
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:346)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> ...
> 2007-06-21 04:02:53,823 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.completeFile: failed to complete /user/hadoopqa/nameNode100Benchmark/output/.214.crc because dir.getFileBlocks() is non-null and pendingFile is null
> 2007-06-21 04:02:53,828 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
> java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:352)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> This last exception occurs over and over again forever (this could be a result of the way NNBench is written).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-1520) IndexOutOfBoundsException in FSEditLog.processIOError

Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12507091 ] 

Nigel Daley commented on HADOOP-1520:
-------------------------------------

It might be significant that the namenode is rolling the fsimage file right before the IndexOutOfBoundsException.

> IndexOutOfBoundsException in FSEditLog.processIOError
> -----------------------------------------------------
>
>                 Key: HADOOP-1520
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1520
>             Project: Hadoop
>          Issue Type: Bug
>    Affects Versions: 0.14.0
>            Reporter: Nigel Daley
>            Assignee: Konstantin Shvachko
>
> Running NNBench I saw these exceptions in the NameNode logs:
> NAMENODE:
> 2007-06-21 04:02:53,587 INFO org.apache.hadoop.fs.FSNamesystem: Roll FSImage
> 2007-06-21 04:02:53,594 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>         at java.util.ArrayList.remove(ArrayList.java:387)
>         at org.apache.hadoop.dfs.FSEditLog.processIOError(FSEditLog.java:169)
>         at org.apache.hadoop.dfs.FSEditLog.logSync(FSEditLog.java:407)
>         at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:878)
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:346)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> ...
> 2007-06-21 04:02:53,823 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.completeFile: failed to complete /user/hadoopqa/nameNode100Benchmark/output/.214.crc because dir.getFileBlocks() is non-null and pendingFile is null
> 2007-06-21 04:02:53,828 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
> java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:352)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> This last exception occurs over and over again forever (this could be a result of the way NNBench is written).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-1520) IndexOutOfBoundsException in FSEditLog.processIOError

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12507565 ] 

dhruba borthakur commented on HADOOP-1520:
------------------------------------------

The DFSClient retries the ClientProtocol.complete() call untill it is successful. I am assuming that a previous complete() call was successfully executed at the NameNode but the client decided to declare that call as "timedout" and retried. The retried call experiences this Exception over and over again.

One generic solution is a have a unique sequence number per RPC that identifies that RPC. The NameNode remembers the last few thousands RPCs that it executed. If a client sends a RPC with a sequence number that exists in the NameNode cache, the NameNode returns the results that were produced by the previous execution of the same RPC.

Can you pl send a pointer to a namenode log file that shows this problem?

> IndexOutOfBoundsException in FSEditLog.processIOError
> -----------------------------------------------------
>
>                 Key: HADOOP-1520
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1520
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.14.0
>            Reporter: Nigel Daley
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.14.0
>
>
> Running NNBench I saw these exceptions in the NameNode logs:
> NAMENODE:
> 2007-06-21 04:02:53,587 INFO org.apache.hadoop.fs.FSNamesystem: Roll FSImage
> 2007-06-21 04:02:53,594 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>         at java.util.ArrayList.remove(ArrayList.java:387)
>         at org.apache.hadoop.dfs.FSEditLog.processIOError(FSEditLog.java:169)
>         at org.apache.hadoop.dfs.FSEditLog.logSync(FSEditLog.java:407)
>         at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:878)
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:346)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> ...
> 2007-06-21 04:02:53,823 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.completeFile: failed to complete /user/hadoopqa/nameNode100Benchmark/output/.214.crc because dir.getFileBlocks() is non-null and pendingFile is null
> 2007-06-21 04:02:53,828 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
> java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:352)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> This last exception occurs over and over again forever (this could be a result of the way NNBench is written).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-1520) IndexOutOfBoundsException in FSEditLog.processIOError

Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

dhruba borthakur updated HADOOP-1520:
-------------------------------------

    Attachment: 1520.patch

I added appropriate synchronization to FSEditLog.java. can you pl see if you started seeing these problems since HADOOP-1003 was applied? Also, if possible, can you apply this patch and rerun the benchmark on the 100 node cluster to see if it fixes your problem? Thanks.

> IndexOutOfBoundsException in FSEditLog.processIOError
> -----------------------------------------------------
>
>                 Key: HADOOP-1520
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1520
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.14.0
>            Reporter: Nigel Daley
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: 1520.patch
>
>
> Running NNBench I saw these exceptions in the NameNode logs:
> NAMENODE:
> 2007-06-21 04:02:53,587 INFO org.apache.hadoop.fs.FSNamesystem: Roll FSImage
> 2007-06-21 04:02:53,594 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>         at java.util.ArrayList.remove(ArrayList.java:387)
>         at org.apache.hadoop.dfs.FSEditLog.processIOError(FSEditLog.java:169)
>         at org.apache.hadoop.dfs.FSEditLog.logSync(FSEditLog.java:407)
>         at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:878)
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:346)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> ...
> 2007-06-21 04:02:53,823 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.completeFile: failed to complete /user/hadoopqa/nameNode100Benchmark/output/.214.crc because dir.getFileBlocks() is non-null and pendingFile is null
> 2007-06-21 04:02:53,828 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
> java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:352)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> This last exception occurs over and over again forever (this could be a result of the way NNBench is written).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HADOOP-1520) IndexOutOfBoundsException in FSEditLog.processIOError

Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nigel Daley updated HADOOP-1520:
--------------------------------

    Fix Version/s: 0.14.0
         Priority: Blocker  (was: Major)

NNBench on 100+ nodes alway hits this problem and hangs.

> IndexOutOfBoundsException in FSEditLog.processIOError
> -----------------------------------------------------
>
>                 Key: HADOOP-1520
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1520
>             Project: Hadoop
>          Issue Type: Bug
>    Affects Versions: 0.14.0
>            Reporter: Nigel Daley
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.14.0
>
>
> Running NNBench I saw these exceptions in the NameNode logs:
> NAMENODE:
> 2007-06-21 04:02:53,587 INFO org.apache.hadoop.fs.FSNamesystem: Roll FSImage
> 2007-06-21 04:02:53,594 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>         at java.util.ArrayList.remove(ArrayList.java:387)
>         at org.apache.hadoop.dfs.FSEditLog.processIOError(FSEditLog.java:169)
>         at org.apache.hadoop.dfs.FSEditLog.logSync(FSEditLog.java:407)
>         at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:878)
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:346)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> ...
> 2007-06-21 04:02:53,823 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.completeFile: failed to complete /user/hadoopqa/nameNode100Benchmark/output/.214.crc because dir.getFileBlocks() is non-null and pendingFile is null
> 2007-06-21 04:02:53,828 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
> java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:352)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> This last exception occurs over and over again forever (this could be a result of the way NNBench is written).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-1520) IndexOutOfBoundsException in FSEditLog.processIOError

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12509008 ] 

Hadoop QA commented on HADOOP-1520:
-----------------------------------

+1

http://issues.apache.org/jira/secure/attachment/12360389/1520.patch applied and successfully tested against trunk revision r551725.

Test results:   http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/347/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/347/console

> IndexOutOfBoundsException in FSEditLog.processIOError
> -----------------------------------------------------
>
>                 Key: HADOOP-1520
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1520
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.14.0
>            Reporter: Nigel Daley
>            Assignee: dhruba borthakur
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: 1520.patch
>
>
> Running NNBench I saw these exceptions in the NameNode logs:
> NAMENODE:
> 2007-06-21 04:02:53,587 INFO org.apache.hadoop.fs.FSNamesystem: Roll FSImage
> 2007-06-21 04:02:53,594 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>         at java.util.ArrayList.remove(ArrayList.java:387)
>         at org.apache.hadoop.dfs.FSEditLog.processIOError(FSEditLog.java:169)
>         at org.apache.hadoop.dfs.FSEditLog.logSync(FSEditLog.java:407)
>         at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:878)
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:346)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> ...
> 2007-06-21 04:02:53,823 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.completeFile: failed to complete /user/hadoopqa/nameNode100Benchmark/output/.214.crc because dir.getFileBlocks() is non-null and pendingFile is null
> 2007-06-21 04:02:53,828 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
> java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:352)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> This last exception occurs over and over again forever (this could be a result of the way NNBench is written).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-1520) IndexOutOfBoundsException in FSEditLog.processIOError

Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12508350 ] 

Nigel Daley commented on HADOOP-1520:
-------------------------------------

+1.  This patch seems to fix the problem.

> IndexOutOfBoundsException in FSEditLog.processIOError
> -----------------------------------------------------
>
>                 Key: HADOOP-1520
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1520
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.14.0
>            Reporter: Nigel Daley
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: 1520.patch
>
>
> Running NNBench I saw these exceptions in the NameNode logs:
> NAMENODE:
> 2007-06-21 04:02:53,587 INFO org.apache.hadoop.fs.FSNamesystem: Roll FSImage
> 2007-06-21 04:02:53,594 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>         at java.util.ArrayList.remove(ArrayList.java:387)
>         at org.apache.hadoop.dfs.FSEditLog.processIOError(FSEditLog.java:169)
>         at org.apache.hadoop.dfs.FSEditLog.logSync(FSEditLog.java:407)
>         at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:878)
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:346)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> ...
> 2007-06-21 04:02:53,823 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.completeFile: failed to complete /user/hadoopqa/nameNode100Benchmark/output/.214.crc because dir.getFileBlocks() is non-null and pendingFile is null
> 2007-06-21 04:02:53,828 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc, DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
> java.io.IOException: Could not complete write to file /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
>         at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:352)
>         at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> This last exception occurs over and over again forever (this could be a result of the way NNBench is written).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.