You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Lohit Vijayarenu (JIRA)" <ji...@apache.org> on 2008/11/20 20:44:44 UTC
[jira] Created: (HADOOP-4697) KFS::getBlockLocations() fails with
files having multiple blocks
KFS::getBlockLocations() fails with files having multiple blocks
----------------------------------------------------------------
Key: HADOOP-4697
URL: https://issues.apache.org/jira/browse/HADOOP-4697
Project: Hadoop Core
Issue Type: Bug
Components: fs
Affects Versions: 0.17.0
Environment: Hadoop on KFS
Reporter: Lohit Vijayarenu
Assignee: Sriram Rao
Fix For: 0.19.1
getBlockLocations() on KFS fail with the following stack trace for large files (with multiple blocks).
{noformat}
java.lang.IllegalArgumentException: Offset 67108864 is outside of file
(0..67108863)
at
org.apache.hadoop.mapred.FileInputFormat.getBlockIndex(FileInputFormat.java:336)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:248)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
at org.apache.hadoop.examples.WordCount.run(WordCount.java:149)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:155)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{noformat}
blkStart was not updated properly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4697) KFS::getBlockLocations() fails with
files having multiple blocks
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649921#action_12649921 ]
Hadoop QA commented on HADOOP-4697:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12394365/patch.4697
against trunk revision 719787.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3630/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3630/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3630/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3630/console
This message is automatically generated.
> KFS::getBlockLocations() fails with files having multiple blocks
> ----------------------------------------------------------------
>
> Key: HADOOP-4697
> URL: https://issues.apache.org/jira/browse/HADOOP-4697
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0
> Environment: Hadoop on KFS
> Reporter: Lohit Vijayarenu
> Assignee: Sriram Rao
> Fix For: 0.19.1
>
> Attachments: patch.4697
>
>
> getBlockLocations() on KFS fail with the following stack trace for large files (with multiple blocks).
> {noformat}
> java.lang.IllegalArgumentException: Offset 67108864 is outside of file
> (0..67108863)
> at
> org.apache.hadoop.mapred.FileInputFormat.getBlockIndex(FileInputFormat.java:336)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:248)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at org.apache.hadoop.examples.WordCount.run(WordCount.java:149)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:54)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
> blkStart was not updated properly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4697) KFS::getBlockLocations() fails with
files having multiple blocks
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-4697:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Sriram
> KFS::getBlockLocations() fails with files having multiple blocks
> ----------------------------------------------------------------
>
> Key: HADOOP-4697
> URL: https://issues.apache.org/jira/browse/HADOOP-4697
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0
> Environment: Hadoop on KFS
> Reporter: Lohit Vijayarenu
> Assignee: Sriram Rao
> Fix For: 0.19.1
>
> Attachments: patch.4697
>
>
> getBlockLocations() on KFS fail with the following stack trace for large files (with multiple blocks).
> {noformat}
> java.lang.IllegalArgumentException: Offset 67108864 is outside of file
> (0..67108863)
> at
> org.apache.hadoop.mapred.FileInputFormat.getBlockIndex(FileInputFormat.java:336)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:248)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at org.apache.hadoop.examples.WordCount.run(WordCount.java:149)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:54)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
> blkStart was not updated properly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4697) KFS::getBlockLocations() fails with
files having multiple blocks
Posted by "Sriram Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sriram Rao updated HADOOP-4697:
-------------------------------
Attachment: patch.4697
A patch that fixes the issue is attached.
No new test is included: this bug was found with a Hadoop+KFS deployment; it was tested in that deployment and verified. Verifying this issue requires a Hadoop+KFS deployment and this is done elsewhere.
> KFS::getBlockLocations() fails with files having multiple blocks
> ----------------------------------------------------------------
>
> Key: HADOOP-4697
> URL: https://issues.apache.org/jira/browse/HADOOP-4697
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.17.0
> Environment: Hadoop on KFS
> Reporter: Lohit Vijayarenu
> Assignee: Sriram Rao
> Fix For: 0.19.1
>
> Attachments: patch.4697
>
>
> getBlockLocations() on KFS fail with the following stack trace for large files (with multiple blocks).
> {noformat}
> java.lang.IllegalArgumentException: Offset 67108864 is outside of file
> (0..67108863)
> at
> org.apache.hadoop.mapred.FileInputFormat.getBlockIndex(FileInputFormat.java:336)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:248)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at org.apache.hadoop.examples.WordCount.run(WordCount.java:149)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:54)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
> blkStart was not updated properly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4697) KFS::getBlockLocations() fails with
files having multiple blocks
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HADOOP-4697:
-------------------------------------
Affects Version/s: (was: 0.17.0)
0.18.0
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
+1. Looks good.
> KFS::getBlockLocations() fails with files having multiple blocks
> ----------------------------------------------------------------
>
> Key: HADOOP-4697
> URL: https://issues.apache.org/jira/browse/HADOOP-4697
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0
> Environment: Hadoop on KFS
> Reporter: Lohit Vijayarenu
> Assignee: Sriram Rao
> Fix For: 0.19.1
>
> Attachments: patch.4697
>
>
> getBlockLocations() on KFS fail with the following stack trace for large files (with multiple blocks).
> {noformat}
> java.lang.IllegalArgumentException: Offset 67108864 is outside of file
> (0..67108863)
> at
> org.apache.hadoop.mapred.FileInputFormat.getBlockIndex(FileInputFormat.java:336)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:248)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at org.apache.hadoop.examples.WordCount.run(WordCount.java:149)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:54)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
> blkStart was not updated properly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4697) KFS::getBlockLocations() fails with
files having multiple blocks
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650670#action_12650670 ]
Hudson commented on HADOOP-4697:
--------------------------------
Integrated in Hadoop-trunk #670 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/670/])
. Fix getBlockLocations in KosmosFileSystem to handle multiple
blocks correctly. Contributed by Sriram Rao.
> KFS::getBlockLocations() fails with files having multiple blocks
> ----------------------------------------------------------------
>
> Key: HADOOP-4697
> URL: https://issues.apache.org/jira/browse/HADOOP-4697
> Project: Hadoop Core
> Issue Type: Bug
> Components: fs
> Affects Versions: 0.18.0
> Environment: Hadoop on KFS
> Reporter: Lohit Vijayarenu
> Assignee: Sriram Rao
> Fix For: 0.19.1
>
> Attachments: patch.4697
>
>
> getBlockLocations() on KFS fail with the following stack trace for large files (with multiple blocks).
> {noformat}
> java.lang.IllegalArgumentException: Offset 67108864 is outside of file
> (0..67108863)
> at
> org.apache.hadoop.mapred.FileInputFormat.getBlockIndex(FileInputFormat.java:336)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:248)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
> at org.apache.hadoop.examples.WordCount.run(WordCount.java:149)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:54)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
> blkStart was not updated properly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.