You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Michael Stack (Jira)" <ji...@apache.org> on 2020/01/13 19:01:00 UTC
[jira] [Commented] (HBASE-23684) NPE HFilesOutputSink
[ https://issues.apache.org/jira/browse/HBASE-23684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014582#comment-17014582 ]
Michael Stack commented on HBASE-23684:
---------------------------------------
Here is another instance:
{code}
2020-01-13 18:43:12,603 INFO org.apache.hadoop.hbase.wal.WALSplitter: Splitting WAL=hdfs://nameservice1/hbase/genie/WALs/hbasedn006.example.org,16020,1578934813898-splitting/hbasedn006.example.org%2C16020%2C1578934813898.1578940951781, size=130.6 M (136916443 bytes)
2020-01-13 18:43:12,605 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recover lease on dfs file hdfs://nameservice1/hbase/genie/WALs/hbasedn006.example.org,16020,1578934813898-splitting/hbasedn006.example.org%2C16020%2C1578934813898.1578940951781
2020-01-13 18:43:12,608 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://nameservice1/hbase/genie/WALs/hbasedn006.example.org,16020,1578934813898-splitting/hbasedn006.example.org%2C16020%2C1578934813898. 1578940951781 after 3ms
2020-01-13 18:43:12,613 INFO org.apache.hadoop.hbase.wal.WALSplitter: Open WAL=hdfs://nameservice1/hbase/genie/WALs/hbasedn006.example.org,16020,1578934813898-splitting/hbasedn006.example.org%2C16020%2C1578934813898.1578940951781 cost 10 ms
2020-01-13 18:43:12,619 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hbasezk002.example.org:2181,hbasezk001.example.org:2181,hbasezk004.example.org:2181,hbasezk003.example.org:2181,hbasezk005.example.org:2181 sessionTimeout=60000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$180/0x00000008005df440@164265a5
2020-01-13 18:43:12,624 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hbasezk003.example.org/17.121.236.6:2181. Will not attempt to authenticate using SASL (unknown error)
2020-01-13 18:43:12,625 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hbasezk003.example.org/17.121.236.6:2181, initiating session
2020-01-13 18:43:12,626 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hbasezk003.example.org/17.121.236.6:2181, sessionid = 0x46dd0ba61659ec7, negotiated timeout = 60000
2020-01-13 18:43:12,856 INFO org.apache.hadoop.hbase.wal.OutputSink: 3 split writer threads finished
2020-01-13 18:43:12,890 INFO org.apache.hadoop.hbase.wal.WALSplitter: Processed 391 edits across 0 regions cost 250 ms; edits skipped=409; WAL=hdfs://nameservice1/hbase/genie/WALs/hbasedn006.example.org,16020,1578934813898-splitting/hbasedn006. example.org%2C16020%2C1578934813898.1578940951781, size=130.6 M, length=136916443, corrupted=false, progress failed=true
2020-01-13 18:43:12,890 WARN org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of WALs/hbasedn006.example.org,16020,1578934813898-splitting/hbasedn006.example.org%2C16020%2C1578934813898.1578940951781 failed, returning error
java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.writeRemainingEntryBuffers(BoundedRecoveredHFilesOutputSink.java:173)
at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.close(BoundedRecoveredHFilesOutputSink.java:140)
at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:339)
at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:181)
at org.apache.hadoop.hbase.regionserver.SplitLogWorker.splitLog(SplitLogWorker.java:105)
at org.apache.hadoop.hbase.regionserver.SplitLogWorker.lambda$new$0(SplitLogWorker.java:84)
at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.configContextForNonMetaWriter(BoundedRecoveredHFilesOutputSink.java:225)
at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.createRecoveredHFileWriter(BoundedRecoveredHFilesOutputSink.java:213)
at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.append(BoundedRecoveredHFilesOutputSink.java:117)
at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.lambda$writeRemainingEntryBuffers$3(BoundedRecoveredHFilesOutputSink.java:155)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
... 3 more
{code}
> NPE HFilesOutputSink
> --------------------
>
> Key: HBASE-23684
> URL: https://issues.apache.org/jira/browse/HBASE-23684
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 2.3.0
> Reporter: Michael Stack
> Priority: Major
>
> Ran into this after enabling hfile splitter:
> {code}
> 2020-01-13 17:37:08,204 INFO org.apache.hadoop.hbase.wal.OutputSink: 3 split writer threads finished
> 2020-01-13 17:37:08,233 INFO org.apache.hadoop.hbase.wal.WALSplitter: Processed 1007 edits across 0 regions cost 284 ms; edits skipped=76; WAL=hdfs://nameservice1/hbase/genie/WALs/hbasedn101.example.org,16020,1578934806382-splitting/hbasedn101.example.org%2C16020%2C1578934806382.1578937008832, size=128.5 M, length=134708720, corrupted=false, progress failed=true
> 2020-01-13 17:37:08,234 WARN org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of WALs/hbasedn101.example.org,16020,1578934806382-splitting/hbasedn101.example.org%2C16020%2C1578934806382.1578937008832 failed, returning error
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.writeRemainingEntryBuffers(BoundedRecoveredHFilesOutputSink.java:173)
> at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.close(BoundedRecoveredHFilesOutputSink.java:140)
> at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:339)
> at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:181)
> at org.apache.hadoop.hbase.regionserver.SplitLogWorker.splitLog(SplitLogWorker.java:105)
> at org.apache.hadoop.hbase.regionserver.SplitLogWorker.lambda$new$0(SplitLogWorker.java:84)
> at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.configContextForNonMetaWriter(BoundedRecoveredHFilesOutputSink.java:225)
> at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.createRecoveredHFileWriter(BoundedRecoveredHFilesOutputSink.java:213)
> at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.append(BoundedRecoveredHFilesOutputSink.java:117)
> at org.apache.hadoop.hbase.wal.BoundedRecoveredHFilesOutputSink.lambda$writeRemainingEntryBuffers$3(BoundedRecoveredHFilesOutputSink.java:155)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> {code}
> It is a bit odd because log says there were zero regions. Not sure what that was about.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)