You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Nichole Treadway (JIRA)" <ji...@apache.org> on 2011/04/14 17:19:05 UTC
[jira] [Created] (HBASE-3782) Multi-Family support for bulk upload
tools causes File Not Found Exception
Multi-Family support for bulk upload tools causes File Not Found Exception
--------------------------------------------------------------------------
Key: HBASE-3782
URL: https://issues.apache.org/jira/browse/HBASE-3782
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 0.90.3
Reporter: Nichole Treadway
I've been testing HBASE-1861 in 0.90.2, which adds multi-family support for bulk upload tools.
I found that when running the importtsv program, some reduce tasks fail with a File Not Found exception if there are no keys in the input data which fall into the region assigned to that reduce task. From what I can determine, it seems that an output directory is created in the write() method and expected to exist in the writeMetaData() method...if there are no keys to be written for that reduce task, the write method is never called and the output directory is never created, but writeMetaData is expecting the output directory to exist...thus the FnF exception:
2011-03-17 11:52:48,095 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
java.io.FileNotFoundException: File does not exist: hdfs://master:9000/awardsData/_temporary/_attempt_201103151859_0066_r_000000_0
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:468)
at org.apache.hadoop.hbase.regionserver.StoreFile.getUniqueFile(StoreFile.java:580)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.writeMetaData(HFileOutputFormat.java:186)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.close(HFileOutputFormat.java:247)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:567)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Simply checking if the file exists should fix the issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3782) Multi-Family support for bulk upload
tools causes File Not Found Exception
Posted by "Nichole Treadway (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nichole Treadway updated HBASE-3782:
------------------------------------
Attachment: HBASE-3782.patch
Just modifying the patch in HBASE-1861
> Multi-Family support for bulk upload tools causes File Not Found Exception
> --------------------------------------------------------------------------
>
> Key: HBASE-3782
> URL: https://issues.apache.org/jira/browse/HBASE-3782
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.90.3
> Reporter: Nichole Treadway
> Attachments: HBASE-3782.patch
>
>
> I've been testing HBASE-1861 in 0.90.2, which adds multi-family support for bulk upload tools.
> I found that when running the importtsv program, some reduce tasks fail with a File Not Found exception if there are no keys in the input data which fall into the region assigned to that reduce task. From what I can determine, it seems that an output directory is created in the write() method and expected to exist in the writeMetaData() method...if there are no keys to be written for that reduce task, the write method is never called and the output directory is never created, but writeMetaData is expecting the output directory to exist...thus the FnF exception:
> 2011-03-17 11:52:48,095 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
> java.io.FileNotFoundException: File does not exist: hdfs://master:9000/awardsData/_temporary/_attempt_201103151859_0066_r_000000_0
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:468)
> at org.apache.hadoop.hbase.regionserver.StoreFile.getUniqueFile(StoreFile.java:580)
> at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.writeMetaData(HFileOutputFormat.java:186)
> at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.close(HFileOutputFormat.java:247)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:567)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Simply checking if the file exists should fix the issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3782) Multi-Family support for bulk
upload tools causes File Not Found Exception
Posted by "Nichole Treadway (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031744#comment-13031744 ]
Nichole Treadway commented on HBASE-3782:
-----------------------------------------
Yes that's right...my patch is modifying the HBASE1861-incomplete.patch.
> Multi-Family support for bulk upload tools causes File Not Found Exception
> --------------------------------------------------------------------------
>
> Key: HBASE-3782
> URL: https://issues.apache.org/jira/browse/HBASE-3782
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.90.3
> Reporter: Nichole Treadway
> Attachments: HBASE-3782.patch
>
>
> I've been testing HBASE-1861 in 0.90.2, which adds multi-family support for bulk upload tools.
> I found that when running the importtsv program, some reduce tasks fail with a File Not Found exception if there are no keys in the input data which fall into the region assigned to that reduce task. From what I can determine, it seems that an output directory is created in the write() method and expected to exist in the writeMetaData() method...if there are no keys to be written for that reduce task, the write method is never called and the output directory is never created, but writeMetaData is expecting the output directory to exist...thus the FnF exception:
> 2011-03-17 11:52:48,095 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
> java.io.FileNotFoundException: File does not exist: hdfs://master:9000/awardsData/_temporary/_attempt_201103151859_0066_r_000000_0
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:468)
> at org.apache.hadoop.hbase.regionserver.StoreFile.getUniqueFile(StoreFile.java:580)
> at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.writeMetaData(HFileOutputFormat.java:186)
> at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.close(HFileOutputFormat.java:247)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:567)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Simply checking if the file exists should fix the issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3782) Multi-Family support for bulk
upload tools causes File Not Found Exception
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029115#comment-13029115 ]
Aaron T. Myers commented on HBASE-3782:
---------------------------------------
I took a look into this. From what I can tell, I believe Nichole's patch modifies the original patch which was posted on HBASE-1861, i.e. {{HBASE1861-incomplete.patch}}. However, what ended up getting committed to resolve HBASE-1861 differs substantially from that original patch. So, I'm pretty confident that this problem isn't applicable to what ended up getting committed.
Nichole, can you confirm this?
> Multi-Family support for bulk upload tools causes File Not Found Exception
> --------------------------------------------------------------------------
>
> Key: HBASE-3782
> URL: https://issues.apache.org/jira/browse/HBASE-3782
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.90.3
> Reporter: Nichole Treadway
> Attachments: HBASE-3782.patch
>
>
> I've been testing HBASE-1861 in 0.90.2, which adds multi-family support for bulk upload tools.
> I found that when running the importtsv program, some reduce tasks fail with a File Not Found exception if there are no keys in the input data which fall into the region assigned to that reduce task. From what I can determine, it seems that an output directory is created in the write() method and expected to exist in the writeMetaData() method...if there are no keys to be written for that reduce task, the write method is never called and the output directory is never created, but writeMetaData is expecting the output directory to exist...thus the FnF exception:
> 2011-03-17 11:52:48,095 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
> java.io.FileNotFoundException: File does not exist: hdfs://master:9000/awardsData/_temporary/_attempt_201103151859_0066_r_000000_0
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:468)
> at org.apache.hadoop.hbase.regionserver.StoreFile.getUniqueFile(StoreFile.java:580)
> at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.writeMetaData(HFileOutputFormat.java:186)
> at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.close(HFileOutputFormat.java:247)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:567)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Simply checking if the file exists should fix the issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3782) Multi-Family support for bulk
upload tools causes File Not Found Exception
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031903#comment-13031903 ]
Aaron T. Myers commented on HBASE-3782:
---------------------------------------
Thanks, Nichole. Can you also confirm that the bug you identified isn't present in the patch which ended up getting committed as part of HBASE-1861? If so, we can close out this issue.
> Multi-Family support for bulk upload tools causes File Not Found Exception
> --------------------------------------------------------------------------
>
> Key: HBASE-3782
> URL: https://issues.apache.org/jira/browse/HBASE-3782
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.90.3
> Reporter: Nichole Treadway
> Attachments: HBASE-3782.patch
>
>
> I've been testing HBASE-1861 in 0.90.2, which adds multi-family support for bulk upload tools.
> I found that when running the importtsv program, some reduce tasks fail with a File Not Found exception if there are no keys in the input data which fall into the region assigned to that reduce task. From what I can determine, it seems that an output directory is created in the write() method and expected to exist in the writeMetaData() method...if there are no keys to be written for that reduce task, the write method is never called and the output directory is never created, but writeMetaData is expecting the output directory to exist...thus the FnF exception:
> 2011-03-17 11:52:48,095 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
> java.io.FileNotFoundException: File does not exist: hdfs://master:9000/awardsData/_temporary/_attempt_201103151859_0066_r_000000_0
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:468)
> at org.apache.hadoop.hbase.regionserver.StoreFile.getUniqueFile(StoreFile.java:580)
> at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.writeMetaData(HFileOutputFormat.java:186)
> at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.close(HFileOutputFormat.java:247)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:567)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Simply checking if the file exists should fix the issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3782) Multi-Family support for bulk
upload tools causes File Not Found Exception
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455436#comment-13455436 ]
Lars Hofhansl commented on HBASE-3782:
--------------------------------------
This is no longer an issue it seems. HFileOutputFormat is now HFile.Writer.appendFileInfo.
Still... Hard to verify.
> Multi-Family support for bulk upload tools causes File Not Found Exception
> --------------------------------------------------------------------------
>
> Key: HBASE-3782
> URL: https://issues.apache.org/jira/browse/HBASE-3782
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.90.3
> Reporter: Nichole Treadway
> Attachments: HBASE-3782.patch
>
>
> I've been testing HBASE-1861 in 0.90.2, which adds multi-family support for bulk upload tools.
> I found that when running the importtsv program, some reduce tasks fail with a File Not Found exception if there are no keys in the input data which fall into the region assigned to that reduce task. From what I can determine, it seems that an output directory is created in the write() method and expected to exist in the writeMetaData() method...if there are no keys to be written for that reduce task, the write method is never called and the output directory is never created, but writeMetaData is expecting the output directory to exist...thus the FnF exception:
> 2011-03-17 11:52:48,095 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
> java.io.FileNotFoundException: File does not exist: hdfs://master:9000/awardsData/_temporary/_attempt_201103151859_0066_r_000000_0
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:468)
> at org.apache.hadoop.hbase.regionserver.StoreFile.getUniqueFile(StoreFile.java:580)
> at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.writeMetaData(HFileOutputFormat.java:186)
> at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.close(HFileOutputFormat.java:247)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:567)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Simply checking if the file exists should fix the issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3782) Multi-Family support for bulk
upload tools causes File Not Found Exception
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019956#comment-13019956 ]
stack commented on HBASE-3782:
------------------------------
Would you mind identifying where the patches differ? Thanks Nichole (I liked 1861 to this one).
> Multi-Family support for bulk upload tools causes File Not Found Exception
> --------------------------------------------------------------------------
>
> Key: HBASE-3782
> URL: https://issues.apache.org/jira/browse/HBASE-3782
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
> Affects Versions: 0.90.3
> Reporter: Nichole Treadway
> Attachments: HBASE-3782.patch
>
>
> I've been testing HBASE-1861 in 0.90.2, which adds multi-family support for bulk upload tools.
> I found that when running the importtsv program, some reduce tasks fail with a File Not Found exception if there are no keys in the input data which fall into the region assigned to that reduce task. From what I can determine, it seems that an output directory is created in the write() method and expected to exist in the writeMetaData() method...if there are no keys to be written for that reduce task, the write method is never called and the output directory is never created, but writeMetaData is expecting the output directory to exist...thus the FnF exception:
> 2011-03-17 11:52:48,095 WARN org.apache.hadoop.mapred.TaskTracker: Error running child
> java.io.FileNotFoundException: File does not exist: hdfs://master:9000/awardsData/_temporary/_attempt_201103151859_0066_r_000000_0
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:468)
> at org.apache.hadoop.hbase.regionserver.StoreFile.getUniqueFile(StoreFile.java:580)
> at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.writeMetaData(HFileOutputFormat.java:186)
> at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat$1.close(HFileOutputFormat.java:247)
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:567)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Simply checking if the file exists should fix the issue.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira