You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2012/07/18 01:20:35 UTC
[jira] [Created] (HBASE-6416) hbck dies on NPE when a region folder
exists but the table does not
Jean-Daniel Cryans created HBASE-6416:
-----------------------------------------
Summary: hbck dies on NPE when a region folder exists but the table does not
Key: HBASE-6416
URL: https://issues.apache.org/jira/browse/HBASE-6416
Project: HBase
Issue Type: Bug
Reporter: Jean-Daniel Cryans
Fix For: 0.96.0, 0.94.2
This is what I'm getting for leftover data that has no .regioninfo
First:
{quote}
12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for region null
java.io.FileNotFoundException: File does not exist: /hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1813)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
at org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{quote}
Then it hangs on:
{quote}
12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir: hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
{quote}
The NPE is sent by:
{code}
Preconditions.checkNotNull("Table " + tableName + "' not present!", tableInfo);
{code}
I wonder why the condition checking was added if we don't handle it... In any case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6416) hbck dies on NPE when a region
folder exists but the table does not
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455771#comment-13455771 ]
Jonathan Hsieh commented on HBASE-6416:
---------------------------------------
I think that having to run hbck multiple times to clean up the corruptions, though not ideal, is acceptable. Ideally the tool will tell the user that they need to do that. At the end of the day, if hbase is down, having a slightly inefficient automatic solution to fix it is better than having no solution to automatically fix it.
> hbck dies on NPE when a region folder exists but the table does not
> -------------------------------------------------------------------
>
> Key: HBASE-6416
> URL: https://issues.apache.org/jira/browse/HBASE-6416
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0, 0.94.3
>
> Attachments: hbase-6416.patch, hbase-6416-v1.patch
>
>
> This is what I'm getting for leftover data that has no .regioninfo
> First:
> {quote}
> 12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for region null
> java.io.FileNotFoundException: File does not exist: /hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
> at org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> Then it hangs on:
> {quote}
> 12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir: hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
> 12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
> at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
> at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
> {quote}
> The NPE is sent by:
> {code}
> Preconditions.checkNotNull("Table " + tableName + "' not present!", tableInfo);
> {code}
> I wonder why the condition checking was added if we don't handle it... In any case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6416) hbck dies on NPE when a region folder
exists but the table does not
Posted by "Jie Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jie Huang updated HBASE-6416:
-----------------------------
Attachment: hbase-6416.patch
Yes. The problem can be reproduced in my test environment. And here attaches one fix for this bug to solve that *hanging around* issue.
According to the jstack result, the most suspicious threads are those "pool-1-thread-*" threads (which are not daemon). I guess it is caused by the following ExecutionException, which leads to an incomplete FutureTask.
{code}
for(int i=0; i<hbiFutures.size(); i++) {
WorkItemHdfsRegionInfo work = hbis.get(i);
Future<Void> f = hbiFutures.get(i);
try {
f.get();
} catch(ExecutionException e) {
LOG.warn("Failed to read .regioninfo file for region " +
work.hbi.getRegionNameAsString(), e.getCause());
}
}
{code}
The simple and quick fix is to shutdown the ExecutorService finally in exec() function.
Now, in my test environment, the process will be terminated with that NPE after applied this patch.
> hbck dies on NPE when a region folder exists but the table does not
> -------------------------------------------------------------------
>
> Key: HBASE-6416
> URL: https://issues.apache.org/jira/browse/HBASE-6416
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0, 0.94.3
>
> Attachments: hbase-6416.patch
>
>
> This is what I'm getting for leftover data that has no .regioninfo
> First:
> {quote}
> 12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for region null
> java.io.FileNotFoundException: File does not exist: /hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
> at org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> Then it hangs on:
> {quote}
> 12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir: hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
> 12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
> at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
> at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
> {quote}
> The NPE is sent by:
> {code}
> Preconditions.checkNotNull("Table " + tableName + "' not present!", tableInfo);
> {code}
> I wonder why the condition checking was added if we don't handle it... In any case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6416) hbck dies on NPE when a region
folder exists but the table does not
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450810#comment-13450810 ]
Jonathan Hsieh commented on HBASE-6416:
---------------------------------------
[~grace.huang] do you want to tackle the other part of the problem as well?
We could probably do the check but instead of using Precondtions, handled it like the dirs == null case where we warn, and return.
> hbck dies on NPE when a region folder exists but the table does not
> -------------------------------------------------------------------
>
> Key: HBASE-6416
> URL: https://issues.apache.org/jira/browse/HBASE-6416
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0, 0.94.3
>
> Attachments: hbase-6416.patch
>
>
> This is what I'm getting for leftover data that has no .regioninfo
> First:
> {quote}
> 12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for region null
> java.io.FileNotFoundException: File does not exist: /hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
> at org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> Then it hangs on:
> {quote}
> 12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir: hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
> 12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
> at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
> at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
> {quote}
> The NPE is sent by:
> {code}
> Preconditions.checkNotNull("Table " + tableName + "' not present!", tableInfo);
> {code}
> I wonder why the condition checking was added if we don't handle it... In any case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6416) hbck dies on NPE when a region
folder exists but the table does not
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451746#comment-13451746 ]
Jonathan Hsieh commented on HBASE-6416:
---------------------------------------
You mean -fixTableOrphans in here instead of -fixHdfsOrphans right?
{code}
- Preconditions.checkNotNull("Table " + tableName + "' not present!", tableInfo);
+ if (tableInfo == null) {
+ LOG.warn("Table '" + tableName + "' not present!");
+ LOG.warn("Try to fix table '" + tableName + "' before -fixHdfsOrphans");
+ return;
+ }
{code}
Also, I'd be fine if 1) the code fixed tableinfo orphans and then fixed region orphans if both option are specified, or 2) if the table orphans are fixed and we warn the user about having to run again using the option. Off the top of my head, I don't know if that is the case. Can you look into that and pick one of the approaches?
> hbck dies on NPE when a region folder exists but the table does not
> -------------------------------------------------------------------
>
> Key: HBASE-6416
> URL: https://issues.apache.org/jira/browse/HBASE-6416
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0, 0.94.3
>
> Attachments: hbase-6416.patch, hbase-6416-v1.patch
>
>
> This is what I'm getting for leftover data that has no .regioninfo
> First:
> {quote}
> 12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for region null
> java.io.FileNotFoundException: File does not exist: /hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
> at org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> Then it hangs on:
> {quote}
> 12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir: hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
> 12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
> at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
> at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
> {quote}
> The NPE is sent by:
> {code}
> Preconditions.checkNotNull("Table " + tableName + "' not present!", tableInfo);
> {code}
> I wonder why the condition checking was added if we don't handle it... In any case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6416) hbck dies on NPE when a region
folder exists but the table does not
Posted by "Jie Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455521#comment-13455521 ]
Jie Huang commented on HBASE-6416:
----------------------------------
Sorry for my late response.
My concern is that now we try to fix .tableinfo file during the online phase, which only happens after the offline phase. However, the fixHdfsOrphan does happen during the offline phase. I wonder if it is possible to recover those missing files respectively. And then, re-run the hbck for a better status.
> hbck dies on NPE when a region folder exists but the table does not
> -------------------------------------------------------------------
>
> Key: HBASE-6416
> URL: https://issues.apache.org/jira/browse/HBASE-6416
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0, 0.94.3
>
> Attachments: hbase-6416.patch, hbase-6416-v1.patch
>
>
> This is what I'm getting for leftover data that has no .regioninfo
> First:
> {quote}
> 12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for region null
> java.io.FileNotFoundException: File does not exist: /hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
> at org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> Then it hangs on:
> {quote}
> 12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir: hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
> 12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
> at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
> at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
> {quote}
> The NPE is sent by:
> {code}
> Preconditions.checkNotNull("Table " + tableName + "' not present!", tableInfo);
> {code}
> I wonder why the condition checking was added if we don't handle it... In any case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6416) hbck dies on NPE when a region
folder exists but the table does not
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482995#comment-13482995 ]
Lars Hofhansl commented on HBASE-6416:
--------------------------------------
[~jmhsieh] Are you saying we should close this?
> hbck dies on NPE when a region folder exists but the table does not
> -------------------------------------------------------------------
>
> Key: HBASE-6416
> URL: https://issues.apache.org/jira/browse/HBASE-6416
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.3, 0.96.0
>
> Attachments: hbase-6416.patch, hbase-6416-v1.patch
>
>
> This is what I'm getting for leftover data that has no .regioninfo
> First:
> {quote}
> 12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for region null
> java.io.FileNotFoundException: File does not exist: /hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
> at org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> Then it hangs on:
> {quote}
> 12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir: hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
> 12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
> at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
> at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
> {quote}
> The NPE is sent by:
> {code}
> Preconditions.checkNotNull("Table " + tableName + "' not present!", tableInfo);
> {code}
> I wonder why the condition checking was added if we don't handle it... In any case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6416) hbck dies on NPE when a region
folder exists but the table does not
Posted by "Jie Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483002#comment-13483002 ]
Jie Huang commented on HBASE-6416:
----------------------------------
I am taking sick leave from Oct. 24 to Oct. 30. Any urgency, please call 18964958151. Sorry for any action delay during these days. Thanks.
> hbck dies on NPE when a region folder exists but the table does not
> -------------------------------------------------------------------
>
> Key: HBASE-6416
> URL: https://issues.apache.org/jira/browse/HBASE-6416
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.3, 0.96.0
>
> Attachments: hbase-6416.patch, hbase-6416-v1.patch
>
>
> This is what I'm getting for leftover data that has no .regioninfo
> First:
> {quote}
> 12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for region null
> java.io.FileNotFoundException: File does not exist: /hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
> at org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> Then it hangs on:
> {quote}
> 12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir: hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
> 12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
> at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
> at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
> {quote}
> The NPE is sent by:
> {code}
> Preconditions.checkNotNull("Table " + tableName + "' not present!", tableInfo);
> {code}
> I wonder why the condition checking was added if we don't handle it... In any case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6416) hbck dies on NPE when a region folder
exists but the table does not
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-6416:
---------------------------------
Fix Version/s: (was: 0.94.2)
0.94.3
> hbck dies on NPE when a region folder exists but the table does not
> -------------------------------------------------------------------
>
> Key: HBASE-6416
> URL: https://issues.apache.org/jira/browse/HBASE-6416
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0, 0.94.3
>
>
> This is what I'm getting for leftover data that has no .regioninfo
> First:
> {quote}
> 12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for region null
> java.io.FileNotFoundException: File does not exist: /hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
> at org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> Then it hangs on:
> {quote}
> 12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir: hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
> 12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
> at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
> at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
> {quote}
> The NPE is sent by:
> {code}
> Preconditions.checkNotNull("Table " + tableName + "' not present!", tableInfo);
> {code}
> I wonder why the condition checking was added if we don't handle it... In any case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6416) hbck dies on NPE when a region folder
exists but the table does not
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-6416:
---------------------------------
Fix Version/s: (was: 0.94.3)
0.94.4
> hbck dies on NPE when a region folder exists but the table does not
> -------------------------------------------------------------------
>
> Key: HBASE-6416
> URL: https://issues.apache.org/jira/browse/HBASE-6416
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0, 0.94.4
>
> Attachments: hbase-6416.patch, hbase-6416-v1.patch
>
>
> This is what I'm getting for leftover data that has no .regioninfo
> First:
> {quote}
> 12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for region null
> java.io.FileNotFoundException: File does not exist: /hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
> at org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> Then it hangs on:
> {quote}
> 12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir: hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
> 12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
> at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
> at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
> {quote}
> The NPE is sent by:
> {code}
> Preconditions.checkNotNull("Table " + tableName + "' not present!", tableInfo);
> {code}
> I wonder why the condition checking was added if we don't handle it... In any case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6416) hbck dies on NPE when a region
folder exists but the table does not
Posted by "Jie Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451588#comment-13451588 ]
Jie Huang commented on HBASE-6416:
----------------------------------
Yes. No problem. I will try to handle the other part too. Thanks for your suggestion. I will upload a new patch file soon.
Meanwhile, I wonder if we need to tell the user to run "-fixOrphanTable" at first, and then re-run "-fixHdfsOrphan" again. Since it can somehow help them repair the system when facing the similar problems.
> hbck dies on NPE when a region folder exists but the table does not
> -------------------------------------------------------------------
>
> Key: HBASE-6416
> URL: https://issues.apache.org/jira/browse/HBASE-6416
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0, 0.94.3
>
> Attachments: hbase-6416.patch
>
>
> This is what I'm getting for leftover data that has no .regioninfo
> First:
> {quote}
> 12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for region null
> java.io.FileNotFoundException: File does not exist: /hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
> at org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> Then it hangs on:
> {quote}
> 12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir: hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
> 12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
> at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
> at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
> {quote}
> The NPE is sent by:
> {code}
> Preconditions.checkNotNull("Table " + tableName + "' not present!", tableInfo);
> {code}
> I wonder why the condition checking was added if we don't handle it... In any case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6416) hbck dies on NPE when a region
folder exists but the table does not
Posted by "Jie Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489947#comment-13489947 ]
Jie Huang commented on HBASE-6416:
----------------------------------
I am taking sick leave from Oct. 31 to Nov. 13. Any urgency, please call 18964958151. Sorry for any action delay during these days. Thanks.
> hbck dies on NPE when a region folder exists but the table does not
> -------------------------------------------------------------------
>
> Key: HBASE-6416
> URL: https://issues.apache.org/jira/browse/HBASE-6416
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0, 0.94.4
>
> Attachments: hbase-6416.patch, hbase-6416-v1.patch
>
>
> This is what I'm getting for leftover data that has no .regioninfo
> First:
> {quote}
> 12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for region null
> java.io.FileNotFoundException: File does not exist: /hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
> at org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> Then it hangs on:
> {quote}
> 12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir: hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
> 12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
> at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
> at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
> {quote}
> The NPE is sent by:
> {code}
> Preconditions.checkNotNull("Table " + tableName + "' not present!", tableInfo);
> {code}
> I wonder why the condition checking was added if we don't handle it... In any case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6416) hbck dies on NPE when a region folder
exists but the table does not
Posted by "Jie Huang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jie Huang updated HBASE-6416:
-----------------------------
Attachment: hbase-6416-v1.patch
Done with quick fix.
> hbck dies on NPE when a region folder exists but the table does not
> -------------------------------------------------------------------
>
> Key: HBASE-6416
> URL: https://issues.apache.org/jira/browse/HBASE-6416
> Project: HBase
> Issue Type: Bug
> Reporter: Jean-Daniel Cryans
> Fix For: 0.96.0, 0.94.3
>
> Attachments: hbase-6416.patch, hbase-6416-v1.patch
>
>
> This is what I'm getting for leftover data that has no .regioninfo
> First:
> {quote}
> 12/07/17 23:13:37 WARN util.HBaseFsck: Failed to read .regioninfo file for region null
> java.io.FileNotFoundException: File does not exist: /hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46/.regioninfo
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1813)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544)
> at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456)
> at org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegioninfo(HBaseFsck.java:611)
> at org.apache.hadoop.hbase.util.HBaseFsck.access$2200(HBaseFsck.java:140)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2882)
> at org.apache.hadoop.hbase.util.HBaseFsck$WorkItemHdfsRegionInfo.call(HBaseFsck.java:2866)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {quote}
> Then it hangs on:
> {quote}
> 12/07/17 23:13:39 INFO util.HBaseFsck: Attempting to handle orphan hdfs dir: hdfs://sfor3s24:10101/hbase/stumble_info_urlid_user/bd5f6cfed674389b4d7b8c1be227cb46
> 12/07/17 23:13:39 INFO util.HBaseFsck: checking orphan for table null
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.HBaseFsck$TableInfo.access$100(HBaseFsck.java:1634)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphan(HBaseFsck.java:435)
> at org.apache.hadoop.hbase.util.HBaseFsck.adoptHdfsOrphans(HBaseFsck.java:408)
> at org.apache.hadoop.hbase.util.HBaseFsck.restoreHdfsIntegrity(HBaseFsck.java:529)
> at org.apache.hadoop.hbase.util.HBaseFsck.offlineHdfsIntegrityRepair(HBaseFsck.java:313)
> at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:386)
> at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3227)
> {quote}
> The NPE is sent by:
> {code}
> Preconditions.checkNotNull("Table " + tableName + "' not present!", tableInfo);
> {code}
> I wonder why the condition checking was added if we don't handle it... In any case hbck dies but it hangs because there are some non-daemon hanging around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira