You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Billy Pearson (JIRA)" <ji...@apache.org> on 2008/01/30 07:20:34 UTC
[jira] Created: (HADOOP-2741) [hbase] Split failed due to spliting
of a split before compaction could complete after first split.
[hbase] Split failed due to spliting of a split before compaction could complete after first split.
---------------------------------------------------------------------------------------------------
Key: HADOOP-2741
URL: https://issues.apache.org/jira/browse/HADOOP-2741
Project: Hadoop Core
Issue Type: Bug
Components: contrib/hbase
Environment: svc hadoop 616618
Reporter: Billy Pearson
Fix For: 0.16.0
This error happened after a region tried to split before a complete compaction could be done on the region from the first split
I thank we should que up a compaction on loading of a region on a region server this should help keep this from happening but also we should have the split thread check to see if the region safe to split.
{code}
2008-01-30 00:07:34,250 ERROR org.apache.hadoop.hbase.HRegionServer: Split failed for region webdata,,1201671207519
java.io.FileNotFoundException: File hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/1080614411/size/mapfiles/5351268197296217146/data does not exist.
at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:341)
at org.apache.hadoop.hbase.HStoreFile.length(HStoreFile.java:456)
at org.apache.hadoop.hbase.HStore.loadHStoreFiles(HStore.java:876)
at org.apache.hadoop.hbase.HStore.<init>(HStore.java:731)
at org.apache.hadoop.hbase.HRegion.<init>(HRegion.java:286)
at org.apache.hadoop.hbase.HRegion.splitRegion(HRegion.java:597)
at org.apache.hadoop.hbase.HRegionServer$Splitter.split(HRegionServer.java:303)
at org.apache.hadoop.hbase.HRegionServer$Splitter.run(HRegionServer.java:263)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2741) [hbase] Split failed due to spliting
of a split before compaction could complete after first split.
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Billy Pearson updated HADOOP-2741:
----------------------------------
Attachment: image1.JPG
> [hbase] Split failed due to spliting of a split before compaction could complete after first split.
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2741
> URL: https://issues.apache.org/jira/browse/HADOOP-2741
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Environment: svc hadoop 616618
> Reporter: Billy Pearson
> Fix For: 0.16.0
>
> Attachments: image1.JPG, image2.JPG
>
>
> This error happened after a region tried to split before a complete compaction could be done on the region from the first split
> I thank we should que up a compaction on loading of a region on a region server this should help keep this from happening but also we should have the split thread check to see if the region safe to split.
> {code}
> 2008-01-30 00:07:34,250 ERROR org.apache.hadoop.hbase.HRegionServer: Split failed for region webdata,,1201671207519
> java.io.FileNotFoundException: File hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/1080614411/size/mapfiles/5351268197296217146/data does not exist.
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:341)
> at org.apache.hadoop.hbase.HStoreFile.length(HStoreFile.java:456)
> at org.apache.hadoop.hbase.HStore.loadHStoreFiles(HStore.java:876)
> at org.apache.hadoop.hbase.HStore.<init>(HStore.java:731)
> at org.apache.hadoop.hbase.HRegion.<init>(HRegion.java:286)
> at org.apache.hadoop.hbase.HRegion.splitRegion(HRegion.java:597)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.split(HRegionServer.java:303)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.run(HRegionServer.java:263)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2741) [hbase] Split failed due to
spliting of a split before compaction could complete after first split.
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564275#action_12564275 ]
Billy Pearson commented on HADOOP-2741:
---------------------------------------
moved to unknown until it can be looked at again I thank it might have to do with the latest patch from HADOOP-2636 so its not related to trunk yet
> [hbase] Split failed due to spliting of a split before compaction could complete after first split.
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2741
> URL: https://issues.apache.org/jira/browse/HADOOP-2741
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Environment: svc hadoop 616618
> Reporter: Billy Pearson
> Attachments: image1.JPG, image2.JPG
>
>
> This error happened after a region tried to split before a complete compaction could be done on the region from the first split
> I thank we should que up a compaction on loading of a region on a region server this should help keep this from happening but also we should have the split thread check to see if the region safe to split.
> {code}
> 2008-01-30 00:07:34,250 ERROR org.apache.hadoop.hbase.HRegionServer: Split failed for region webdata,,1201671207519
> java.io.FileNotFoundException: File hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/1080614411/size/mapfiles/5351268197296217146/data does not exist.
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:341)
> at org.apache.hadoop.hbase.HStoreFile.length(HStoreFile.java:456)
> at org.apache.hadoop.hbase.HStore.loadHStoreFiles(HStore.java:876)
> at org.apache.hadoop.hbase.HStore.<init>(HStore.java:731)
> at org.apache.hadoop.hbase.HRegion.<init>(HRegion.java:286)
> at org.apache.hadoop.hbase.HRegion.splitRegion(HRegion.java:597)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.split(HRegionServer.java:303)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.run(HRegionServer.java:263)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2741) [hbase] Split failed due to
spliting of a split before compaction could complete after first split.
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12563848#action_12563848 ]
Billy Pearson commented on HADOOP-2741:
---------------------------------------
The region server does not crash from this but it does take the region offline and it never comes back online.
> [hbase] Split failed due to spliting of a split before compaction could complete after first split.
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2741
> URL: https://issues.apache.org/jira/browse/HADOOP-2741
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Environment: svc hadoop 616618
> Reporter: Billy Pearson
> Fix For: 0.16.0
>
>
> This error happened after a region tried to split before a complete compaction could be done on the region from the first split
> I thank we should que up a compaction on loading of a region on a region server this should help keep this from happening but also we should have the split thread check to see if the region safe to split.
> {code}
> 2008-01-30 00:07:34,250 ERROR org.apache.hadoop.hbase.HRegionServer: Split failed for region webdata,,1201671207519
> java.io.FileNotFoundException: File hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/1080614411/size/mapfiles/5351268197296217146/data does not exist.
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:341)
> at org.apache.hadoop.hbase.HStoreFile.length(HStoreFile.java:456)
> at org.apache.hadoop.hbase.HStore.loadHStoreFiles(HStore.java:876)
> at org.apache.hadoop.hbase.HStore.<init>(HStore.java:731)
> at org.apache.hadoop.hbase.HRegion.<init>(HRegion.java:286)
> at org.apache.hadoop.hbase.HRegion.splitRegion(HRegion.java:597)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.split(HRegionServer.java:303)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.run(HRegionServer.java:263)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2741) [hbase] Split failed due to
spliting of a split before compaction could complete after first split.
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12563860#action_12563860 ]
Billy Pearson commented on HADOOP-2741:
---------------------------------------
I thought this was from a call to a file split but looking now and see what failed somehow the file just is not there.
I was testing the splits to see if the split would fail if all family columns where not compacted before the next split so we would have to reference source file 2 split levels but not sure if this is the bug that did the above error or not.
> [hbase] Split failed due to spliting of a split before compaction could complete after first split.
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2741
> URL: https://issues.apache.org/jira/browse/HADOOP-2741
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Environment: svc hadoop 616618
> Reporter: Billy Pearson
> Fix For: 0.16.0
>
>
> This error happened after a region tried to split before a complete compaction could be done on the region from the first split
> I thank we should que up a compaction on loading of a region on a region server this should help keep this from happening but also we should have the split thread check to see if the region safe to split.
> {code}
> 2008-01-30 00:07:34,250 ERROR org.apache.hadoop.hbase.HRegionServer: Split failed for region webdata,,1201671207519
> java.io.FileNotFoundException: File hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/1080614411/size/mapfiles/5351268197296217146/data does not exist.
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:341)
> at org.apache.hadoop.hbase.HStoreFile.length(HStoreFile.java:456)
> at org.apache.hadoop.hbase.HStore.loadHStoreFiles(HStore.java:876)
> at org.apache.hadoop.hbase.HStore.<init>(HStore.java:731)
> at org.apache.hadoop.hbase.HRegion.<init>(HRegion.java:286)
> at org.apache.hadoop.hbase.HRegion.splitRegion(HRegion.java:597)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.split(HRegionServer.java:303)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.run(HRegionServer.java:263)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-2741) [hbase] Split failed due to spliting
of a split before compaction could complete after first split.
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Billy Pearson resolved HADOOP-2741.
-----------------------------------
Resolution: Cannot Reproduce
I thank a combine commit of several patches make this issue go away
> [hbase] Split failed due to spliting of a split before compaction could complete after first split.
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2741
> URL: https://issues.apache.org/jira/browse/HADOOP-2741
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Environment: svc hadoop 616618
> Reporter: Billy Pearson
> Attachments: image1.JPG, image2.JPG
>
>
> This error happened after a region tried to split before a complete compaction could be done on the region from the first split
> I thank we should que up a compaction on loading of a region on a region server this should help keep this from happening but also we should have the split thread check to see if the region safe to split.
> {code}
> 2008-01-30 00:07:34,250 ERROR org.apache.hadoop.hbase.HRegionServer: Split failed for region webdata,,1201671207519
> java.io.FileNotFoundException: File hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/1080614411/size/mapfiles/5351268197296217146/data does not exist.
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:341)
> at org.apache.hadoop.hbase.HStoreFile.length(HStoreFile.java:456)
> at org.apache.hadoop.hbase.HStore.loadHStoreFiles(HStore.java:876)
> at org.apache.hadoop.hbase.HStore.<init>(HStore.java:731)
> at org.apache.hadoop.hbase.HRegion.<init>(HRegion.java:286)
> at org.apache.hadoop.hbase.HRegion.splitRegion(HRegion.java:597)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.split(HRegionServer.java:303)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.run(HRegionServer.java:263)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2741) [hbase] Split failed due to
spliting of a split before compaction could complete after first split.
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12563863#action_12563863 ]
Billy Pearson commented on HADOOP-2741:
---------------------------------------
attached images
image1 is the
/gfs_storage/hadoop-root/hbase/webdata/1080614411/size/mapfiles/
showing the reference for the split
image2 shoes the reference folder
> [hbase] Split failed due to spliting of a split before compaction could complete after first split.
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2741
> URL: https://issues.apache.org/jira/browse/HADOOP-2741
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Environment: svc hadoop 616618
> Reporter: Billy Pearson
> Fix For: 0.16.0
>
> Attachments: image1.JPG, image2.JPG
>
>
> This error happened after a region tried to split before a complete compaction could be done on the region from the first split
> I thank we should que up a compaction on loading of a region on a region server this should help keep this from happening but also we should have the split thread check to see if the region safe to split.
> {code}
> 2008-01-30 00:07:34,250 ERROR org.apache.hadoop.hbase.HRegionServer: Split failed for region webdata,,1201671207519
> java.io.FileNotFoundException: File hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/1080614411/size/mapfiles/5351268197296217146/data does not exist.
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:341)
> at org.apache.hadoop.hbase.HStoreFile.length(HStoreFile.java:456)
> at org.apache.hadoop.hbase.HStore.loadHStoreFiles(HStore.java:876)
> at org.apache.hadoop.hbase.HStore.<init>(HStore.java:731)
> at org.apache.hadoop.hbase.HRegion.<init>(HRegion.java:286)
> at org.apache.hadoop.hbase.HRegion.splitRegion(HRegion.java:597)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.split(HRegionServer.java:303)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.run(HRegionServer.java:263)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2741) [hbase] Split failed due to
spliting of a split before compaction could complete after first split.
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12564251#action_12564251 ]
Mukund Madhugiri commented on HADOOP-2741:
------------------------------------------
This is marked for 0.16.0. Is it a blocker for 0.16.0 (not currently marked as a blocker) or can it be moved to 0.16.1?
Thanks
> [hbase] Split failed due to spliting of a split before compaction could complete after first split.
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2741
> URL: https://issues.apache.org/jira/browse/HADOOP-2741
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Environment: svc hadoop 616618
> Reporter: Billy Pearson
> Fix For: 0.16.0
>
> Attachments: image1.JPG, image2.JPG
>
>
> This error happened after a region tried to split before a complete compaction could be done on the region from the first split
> I thank we should que up a compaction on loading of a region on a region server this should help keep this from happening but also we should have the split thread check to see if the region safe to split.
> {code}
> 2008-01-30 00:07:34,250 ERROR org.apache.hadoop.hbase.HRegionServer: Split failed for region webdata,,1201671207519
> java.io.FileNotFoundException: File hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/1080614411/size/mapfiles/5351268197296217146/data does not exist.
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:341)
> at org.apache.hadoop.hbase.HStoreFile.length(HStoreFile.java:456)
> at org.apache.hadoop.hbase.HStore.loadHStoreFiles(HStore.java:876)
> at org.apache.hadoop.hbase.HStore.<init>(HStore.java:731)
> at org.apache.hadoop.hbase.HRegion.<init>(HRegion.java:286)
> at org.apache.hadoop.hbase.HRegion.splitRegion(HRegion.java:597)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.split(HRegionServer.java:303)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.run(HRegionServer.java:263)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2741) [hbase] Split failed due to spliting
of a split before compaction could complete after first split.
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Billy Pearson updated HADOOP-2741:
----------------------------------
Fix Version/s: (was: 0.16.0)
> [hbase] Split failed due to spliting of a split before compaction could complete after first split.
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2741
> URL: https://issues.apache.org/jira/browse/HADOOP-2741
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Environment: svc hadoop 616618
> Reporter: Billy Pearson
> Attachments: image1.JPG, image2.JPG
>
>
> This error happened after a region tried to split before a complete compaction could be done on the region from the first split
> I thank we should que up a compaction on loading of a region on a region server this should help keep this from happening but also we should have the split thread check to see if the region safe to split.
> {code}
> 2008-01-30 00:07:34,250 ERROR org.apache.hadoop.hbase.HRegionServer: Split failed for region webdata,,1201671207519
> java.io.FileNotFoundException: File hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/1080614411/size/mapfiles/5351268197296217146/data does not exist.
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:341)
> at org.apache.hadoop.hbase.HStoreFile.length(HStoreFile.java:456)
> at org.apache.hadoop.hbase.HStore.loadHStoreFiles(HStore.java:876)
> at org.apache.hadoop.hbase.HStore.<init>(HStore.java:731)
> at org.apache.hadoop.hbase.HRegion.<init>(HRegion.java:286)
> at org.apache.hadoop.hbase.HRegion.splitRegion(HRegion.java:597)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.split(HRegionServer.java:303)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.run(HRegionServer.java:263)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2741) [hbase] Split failed due to spliting
of a split before compaction could complete after first split.
Posted by "Billy Pearson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Billy Pearson updated HADOOP-2741:
----------------------------------
Attachment: image2.JPG
> [hbase] Split failed due to spliting of a split before compaction could complete after first split.
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2741
> URL: https://issues.apache.org/jira/browse/HADOOP-2741
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/hbase
> Environment: svc hadoop 616618
> Reporter: Billy Pearson
> Fix For: 0.16.0
>
> Attachments: image1.JPG, image2.JPG
>
>
> This error happened after a region tried to split before a complete compaction could be done on the region from the first split
> I thank we should que up a compaction on loading of a region on a region server this should help keep this from happening but also we should have the split thread check to see if the region safe to split.
> {code}
> 2008-01-30 00:07:34,250 ERROR org.apache.hadoop.hbase.HRegionServer: Split failed for region webdata,,1201671207519
> java.io.FileNotFoundException: File hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/webdata/1080614411/size/mapfiles/5351268197296217146/data does not exist.
> at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:341)
> at org.apache.hadoop.hbase.HStoreFile.length(HStoreFile.java:456)
> at org.apache.hadoop.hbase.HStore.loadHStoreFiles(HStore.java:876)
> at org.apache.hadoop.hbase.HStore.<init>(HStore.java:731)
> at org.apache.hadoop.hbase.HRegion.<init>(HRegion.java:286)
> at org.apache.hadoop.hbase.HRegion.splitRegion(HRegion.java:597)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.split(HRegionServer.java:303)
> at org.apache.hadoop.hbase.HRegionServer$Splitter.run(HRegionServer.java:263)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.