You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Prakash Khemani (JIRA)" <ji...@apache.org> on 2010/11/16 21:22:57 UTC
[jira] Created: (HBASE-3239) NPE when trying to roll logs
NPE when trying to roll logs
----------------------------
Key: HBASE-3239
URL: https://issues.apache.org/jira/browse/HBASE-3239
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.90.0
Reporter: Prakash Khemani
Note from Kannan
findMemstoresWithEditsEqualOrOlderThan() can return NULL it seems like. And we don't check NULL, before "region.length".
regions = findMemstoresWithEditsEqualOrOlderThan(this.outputfiles.firstKey(),
this.lastSeqWritten);
StringBuilder sb = new StringBuilder();
for (int i = 0; i < regions.length; i++) {
===
Stack Trace
2010-11-15 19:19:54,258 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.1 GB, free=1.71 GB, max=7.81 GB, blocks=385740, accesses=7020255, hits=6329399, hitRatio=90.15%%, cachingAccesses=6765050, cachingHits=6329399, cachingHitsRatio=93.56%%, evictions=1, evicted=49911, evictedPerRun=49911.0
2010-11-15 19:21:05,204 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
2010-11-15 19:21:05,211 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877154987, entries=649004, filesize=255069060. New hlog /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877665062
2010-11-15 19:21:05,222 ERROR org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
2010-11-15 19:21:05,226 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=pumahbase042.snc5.facebook.com,60020,1289856892583, load=(requests=3476, regions=40, usedHeap=8388, maxHeap=15987): Log rolling failed
java.lang.NullPointerException
at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=1264.5834, regions=40, stores=70, storefiles=98, storefileIndexSize=35, memstoreSize=83, compactionQueueSize=0, usedHeap=8370, maxHeap=15987, blockCacheSize=6593768536, blockCacheFree=1788154792, blockCacheCount=388283, blockCacheHitRatio=90, blockCacheHitCachingRatio=93
2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Log rolling failed
2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
2010-11-15 19:21:07,255 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
===
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3239) NPE when trying to roll logs
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932766#action_12932766 ]
Kannan Muthukkaruppan commented on HBASE-3239:
----------------------------------------------
While the fix seems straightforward, i.e. add a safety check before going into the loop, I still haven't been able to explain how this case arises, namely, that we have a lot of log files accumulated, but we can't find a single region/memstore that contains some edits present in the oldest log.
> NPE when trying to roll logs
> ----------------------------
>
> Key: HBASE-3239
> URL: https://issues.apache.org/jira/browse/HBASE-3239
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0
> Reporter: Prakash Khemani
>
> Note from Kannan
> findMemstoresWithEditsEqualOrOlderThan() can return NULL it seems like. And we don't check NULL, before "region.length".
> regions = findMemstoresWithEditsEqualOrOlderThan(this.outputfiles.firstKey(),
> this.lastSeqWritten);
> StringBuilder sb = new StringBuilder();
> for (int i = 0; i < regions.length; i++) {
> ===
> Stack Trace
> 2010-11-15 19:19:54,258 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.1 GB, free=1.71 GB, max=7.81 GB, blocks=385740, accesses=7020255, hits=6329399, hitRatio=90.15%%, cachingAccesses=6765050, cachingHits=6329399, cachingHitsRatio=93.56%%, evictions=1, evicted=49911, evictedPerRun=49911.0
> 2010-11-15 19:21:05,204 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2010-11-15 19:21:05,211 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877154987, entries=649004, filesize=255069060. New hlog /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877665062
> 2010-11-15 19:21:05,222 ERROR org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,226 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=pumahbase042.snc5.facebook.com,60020,1289856892583, load=(requests=3476, regions=40, usedHeap=8388, maxHeap=15987): Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=1264.5834, regions=40, stores=70, storefiles=98, storefileIndexSize=35, memstoreSize=83, compactionQueueSize=0, usedHeap=8370, maxHeap=15987, blockCacheSize=6593768536, blockCacheFree=1788154792, blockCacheCount=388283, blockCacheHitRatio=90, blockCacheHitCachingRatio=93
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Log rolling failed
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
> 2010-11-15 19:21:07,255 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> ===
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3239) NPE when trying to roll logs
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932965#action_12932965 ]
Jonathan Gray commented on HBASE-3239:
--------------------------------------
@Kannan, I think the case you're describing and what is seen here is a fairly "normal" behavior, though it may not be seen often in practice if you have good distribution of writes across regions.
Two possible but maybe not common/real world examples:
- All edits in the oldest log are for regions that have moved to other servers
- You have 4 regions, flush size is 64MB, hlog size is 64MB, you hold max 10 logs. With even distribution of writes, you would expect only the most recent 4 logs (latest 256MB of writes) to have any data which is present in the current memstores (maximum total size of 64*4=256MB). Eviction of the oldest logs will not contain any edits that are in the memstores of these regions.
> NPE when trying to roll logs
> ----------------------------
>
> Key: HBASE-3239
> URL: https://issues.apache.org/jira/browse/HBASE-3239
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0
> Reporter: Prakash Khemani
> Priority: Blocker
> Fix For: 0.90.0
>
>
> Note from Kannan
> findMemstoresWithEditsEqualOrOlderThan() can return NULL it seems like. And we don't check NULL, before "region.length".
> regions = findMemstoresWithEditsEqualOrOlderThan(this.outputfiles.firstKey(),
> this.lastSeqWritten);
> StringBuilder sb = new StringBuilder();
> for (int i = 0; i < regions.length; i++) {
> ===
> Stack Trace
> 2010-11-15 19:19:54,258 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.1 GB, free=1.71 GB, max=7.81 GB, blocks=385740, accesses=7020255, hits=6329399, hitRatio=90.15%%, cachingAccesses=6765050, cachingHits=6329399, cachingHitsRatio=93.56%%, evictions=1, evicted=49911, evictedPerRun=49911.0
> 2010-11-15 19:21:05,204 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2010-11-15 19:21:05,211 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877154987, entries=649004, filesize=255069060. New hlog /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877665062
> 2010-11-15 19:21:05,222 ERROR org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,226 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=pumahbase042.snc5.facebook.com,60020,1289856892583, load=(requests=3476, regions=40, usedHeap=8388, maxHeap=15987): Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=1264.5834, regions=40, stores=70, storefiles=98, storefileIndexSize=35, memstoreSize=83, compactionQueueSize=0, usedHeap=8370, maxHeap=15987, blockCacheSize=6593768536, blockCacheFree=1788154792, blockCacheCount=388283, blockCacheHitRatio=90, blockCacheHitCachingRatio=93
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Log rolling failed
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
> 2010-11-15 19:21:07,255 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> ===
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3239) NPE when trying to roll logs
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933003#action_12933003 ]
Kannan Muthukkaruppan commented on HBASE-3239:
----------------------------------------------
The patch uploaded as part of HBASE-3241 also contains a fix for this issue.
> NPE when trying to roll logs
> ----------------------------
>
> Key: HBASE-3239
> URL: https://issues.apache.org/jira/browse/HBASE-3239
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0
> Reporter: Prakash Khemani
> Assignee: Kannan Muthukkaruppan
> Priority: Blocker
> Fix For: 0.90.0
>
>
> Note from Kannan
> findMemstoresWithEditsEqualOrOlderThan() can return NULL it seems like. And we don't check NULL, before "region.length".
> regions = findMemstoresWithEditsEqualOrOlderThan(this.outputfiles.firstKey(),
> this.lastSeqWritten);
> StringBuilder sb = new StringBuilder();
> for (int i = 0; i < regions.length; i++) {
> ===
> Stack Trace
> 2010-11-15 19:19:54,258 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.1 GB, free=1.71 GB, max=7.81 GB, blocks=385740, accesses=7020255, hits=6329399, hitRatio=90.15%%, cachingAccesses=6765050, cachingHits=6329399, cachingHitsRatio=93.56%%, evictions=1, evicted=49911, evictedPerRun=49911.0
> 2010-11-15 19:21:05,204 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2010-11-15 19:21:05,211 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877154987, entries=649004, filesize=255069060. New hlog /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877665062
> 2010-11-15 19:21:05,222 ERROR org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,226 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=pumahbase042.snc5.facebook.com,60020,1289856892583, load=(requests=3476, regions=40, usedHeap=8388, maxHeap=15987): Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=1264.5834, regions=40, stores=70, storefiles=98, storefileIndexSize=35, memstoreSize=83, compactionQueueSize=0, usedHeap=8370, maxHeap=15987, blockCacheSize=6593768536, blockCacheFree=1788154792, blockCacheCount=388283, blockCacheHitRatio=90, blockCacheHitCachingRatio=93
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Log rolling failed
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
> 2010-11-15 19:21:07,255 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> ===
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3239) Handle null regions to flush in
HLog.cleanOldLogs
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-3239:
--------------------------------------
Summary: Handle null regions to flush in HLog.cleanOldLogs (was: NPE when trying to roll logs)
About to commit, changing the title.
> Handle null regions to flush in HLog.cleanOldLogs
> -------------------------------------------------
>
> Key: HBASE-3239
> URL: https://issues.apache.org/jira/browse/HBASE-3239
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0
> Reporter: Prakash Khemani
> Assignee: Kannan Muthukkaruppan
> Priority: Blocker
> Fix For: 0.90.0
>
>
> Note from Kannan
> findMemstoresWithEditsEqualOrOlderThan() can return NULL it seems like. And we don't check NULL, before "region.length".
> regions = findMemstoresWithEditsEqualOrOlderThan(this.outputfiles.firstKey(),
> this.lastSeqWritten);
> StringBuilder sb = new StringBuilder();
> for (int i = 0; i < regions.length; i++) {
> ===
> Stack Trace
> 2010-11-15 19:19:54,258 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.1 GB, free=1.71 GB, max=7.81 GB, blocks=385740, accesses=7020255, hits=6329399, hitRatio=90.15%%, cachingAccesses=6765050, cachingHits=6329399, cachingHitsRatio=93.56%%, evictions=1, evicted=49911, evictedPerRun=49911.0
> 2010-11-15 19:21:05,204 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2010-11-15 19:21:05,211 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877154987, entries=649004, filesize=255069060. New hlog /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877665062
> 2010-11-15 19:21:05,222 ERROR org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,226 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=pumahbase042.snc5.facebook.com,60020,1289856892583, load=(requests=3476, regions=40, usedHeap=8388, maxHeap=15987): Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=1264.5834, regions=40, stores=70, storefiles=98, storefileIndexSize=35, memstoreSize=83, compactionQueueSize=0, usedHeap=8370, maxHeap=15987, blockCacheSize=6593768536, blockCacheFree=1788154792, blockCacheCount=388283, blockCacheHitRatio=90, blockCacheHitCachingRatio=93
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Log rolling failed
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
> 2010-11-15 19:21:07,255 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> ===
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-3239) NPE when trying to roll logs
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray reassigned HBASE-3239:
------------------------------------
Assignee: Kannan Muthukkaruppan
Assigning to Kannan who made a fix for this in our internal branch.
> NPE when trying to roll logs
> ----------------------------
>
> Key: HBASE-3239
> URL: https://issues.apache.org/jira/browse/HBASE-3239
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0
> Reporter: Prakash Khemani
> Assignee: Kannan Muthukkaruppan
> Priority: Blocker
> Fix For: 0.90.0
>
>
> Note from Kannan
> findMemstoresWithEditsEqualOrOlderThan() can return NULL it seems like. And we don't check NULL, before "region.length".
> regions = findMemstoresWithEditsEqualOrOlderThan(this.outputfiles.firstKey(),
> this.lastSeqWritten);
> StringBuilder sb = new StringBuilder();
> for (int i = 0; i < regions.length; i++) {
> ===
> Stack Trace
> 2010-11-15 19:19:54,258 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.1 GB, free=1.71 GB, max=7.81 GB, blocks=385740, accesses=7020255, hits=6329399, hitRatio=90.15%%, cachingAccesses=6765050, cachingHits=6329399, cachingHitsRatio=93.56%%, evictions=1, evicted=49911, evictedPerRun=49911.0
> 2010-11-15 19:21:05,204 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2010-11-15 19:21:05,211 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877154987, entries=649004, filesize=255069060. New hlog /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877665062
> 2010-11-15 19:21:05,222 ERROR org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,226 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=pumahbase042.snc5.facebook.com,60020,1289856892583, load=(requests=3476, regions=40, usedHeap=8388, maxHeap=15987): Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=1264.5834, regions=40, stores=70, storefiles=98, storefileIndexSize=35, memstoreSize=83, compactionQueueSize=0, usedHeap=8370, maxHeap=15987, blockCacheSize=6593768536, blockCacheFree=1788154792, blockCacheCount=388283, blockCacheHitRatio=90, blockCacheHitCachingRatio=93
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Log rolling failed
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
> 2010-11-15 19:21:07,255 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> ===
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3239) NPE when trying to roll logs
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-3239:
---------------------------------
Priority: Blocker (was: Major)
Fix Version/s: 0.90.0
Making blocker against 0.90
> NPE when trying to roll logs
> ----------------------------
>
> Key: HBASE-3239
> URL: https://issues.apache.org/jira/browse/HBASE-3239
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0
> Reporter: Prakash Khemani
> Priority: Blocker
> Fix For: 0.90.0
>
>
> Note from Kannan
> findMemstoresWithEditsEqualOrOlderThan() can return NULL it seems like. And we don't check NULL, before "region.length".
> regions = findMemstoresWithEditsEqualOrOlderThan(this.outputfiles.firstKey(),
> this.lastSeqWritten);
> StringBuilder sb = new StringBuilder();
> for (int i = 0; i < regions.length; i++) {
> ===
> Stack Trace
> 2010-11-15 19:19:54,258 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.1 GB, free=1.71 GB, max=7.81 GB, blocks=385740, accesses=7020255, hits=6329399, hitRatio=90.15%%, cachingAccesses=6765050, cachingHits=6329399, cachingHitsRatio=93.56%%, evictions=1, evicted=49911, evictedPerRun=49911.0
> 2010-11-15 19:21:05,204 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2010-11-15 19:21:05,211 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877154987, entries=649004, filesize=255069060. New hlog /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877665062
> 2010-11-15 19:21:05,222 ERROR org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,226 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=pumahbase042.snc5.facebook.com,60020,1289856892583, load=(requests=3476, regions=40, usedHeap=8388, maxHeap=15987): Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=1264.5834, regions=40, stores=70, storefiles=98, storefileIndexSize=35, memstoreSize=83, compactionQueueSize=0, usedHeap=8370, maxHeap=15987, blockCacheSize=6593768536, blockCacheFree=1788154792, blockCacheCount=388283, blockCacheHitRatio=90, blockCacheHitCachingRatio=93
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Log rolling failed
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
> 2010-11-15 19:21:07,255 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> ===
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3239) NPE when trying to roll logs
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933191#action_12933191 ]
Kannan Muthukkaruppan commented on HBASE-3239:
----------------------------------------------
Note: The cluster this happened on didn't have HBASE-3208 & HBASE-3198 fixes.
> NPE when trying to roll logs
> ----------------------------
>
> Key: HBASE-3239
> URL: https://issues.apache.org/jira/browse/HBASE-3239
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0
> Reporter: Prakash Khemani
> Assignee: Kannan Muthukkaruppan
> Priority: Blocker
> Fix For: 0.90.0
>
>
> Note from Kannan
> findMemstoresWithEditsEqualOrOlderThan() can return NULL it seems like. And we don't check NULL, before "region.length".
> regions = findMemstoresWithEditsEqualOrOlderThan(this.outputfiles.firstKey(),
> this.lastSeqWritten);
> StringBuilder sb = new StringBuilder();
> for (int i = 0; i < regions.length; i++) {
> ===
> Stack Trace
> 2010-11-15 19:19:54,258 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.1 GB, free=1.71 GB, max=7.81 GB, blocks=385740, accesses=7020255, hits=6329399, hitRatio=90.15%%, cachingAccesses=6765050, cachingHits=6329399, cachingHitsRatio=93.56%%, evictions=1, evicted=49911, evictedPerRun=49911.0
> 2010-11-15 19:21:05,204 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2010-11-15 19:21:05,211 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877154987, entries=649004, filesize=255069060. New hlog /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877665062
> 2010-11-15 19:21:05,222 ERROR org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,226 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=pumahbase042.snc5.facebook.com,60020,1289856892583, load=(requests=3476, regions=40, usedHeap=8388, maxHeap=15987): Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=1264.5834, regions=40, stores=70, storefiles=98, storefileIndexSize=35, memstoreSize=83, compactionQueueSize=0, usedHeap=8370, maxHeap=15987, blockCacheSize=6593768536, blockCacheFree=1788154792, blockCacheCount=388283, blockCacheHitRatio=90, blockCacheHitCachingRatio=93
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Log rolling failed
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
> 2010-11-15 19:21:07,255 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> ===
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-3239) Handle null regions to flush in
HLog.cleanOldLogs
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans resolved HBASE-3239.
---------------------------------------
Resolution: Fixed
Fix Version/s: 0.92.0
Hadoop Flags: [Reviewed]
Committed to trunk and 0.90, thanks Kannan!
> Handle null regions to flush in HLog.cleanOldLogs
> -------------------------------------------------
>
> Key: HBASE-3239
> URL: https://issues.apache.org/jira/browse/HBASE-3239
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0
> Reporter: Prakash Khemani
> Assignee: Kannan Muthukkaruppan
> Priority: Blocker
> Fix For: 0.90.0, 0.92.0
>
>
> Note from Kannan
> findMemstoresWithEditsEqualOrOlderThan() can return NULL it seems like. And we don't check NULL, before "region.length".
> regions = findMemstoresWithEditsEqualOrOlderThan(this.outputfiles.firstKey(),
> this.lastSeqWritten);
> StringBuilder sb = new StringBuilder();
> for (int i = 0; i < regions.length; i++) {
> ===
> Stack Trace
> 2010-11-15 19:19:54,258 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.1 GB, free=1.71 GB, max=7.81 GB, blocks=385740, accesses=7020255, hits=6329399, hitRatio=90.15%%, cachingAccesses=6765050, cachingHits=6329399, cachingHitsRatio=93.56%%, evictions=1, evicted=49911, evictedPerRun=49911.0
> 2010-11-15 19:21:05,204 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2010-11-15 19:21:05,211 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877154987, entries=649004, filesize=255069060. New hlog /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877665062
> 2010-11-15 19:21:05,222 ERROR org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,226 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=pumahbase042.snc5.facebook.com,60020,1289856892583, load=(requests=3476, regions=40, usedHeap=8388, maxHeap=15987): Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=1264.5834, regions=40, stores=70, storefiles=98, storefileIndexSize=35, memstoreSize=83, compactionQueueSize=0, usedHeap=8370, maxHeap=15987, blockCacheSize=6593768536, blockCacheFree=1788154792, blockCacheCount=388283, blockCacheHitRatio=90, blockCacheHitCachingRatio=93
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Log rolling failed
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
> 2010-11-15 19:21:07,255 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> ===
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3239) NPE when trying to roll logs
Posted by "Kannan Muthukkaruppan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933144#action_12933144 ]
Kannan Muthukkaruppan commented on HBASE-3239:
----------------------------------------------
Jonathan/JD: Still not sure about the examples. So my understanding is as follows.
cleanOldLogs() has two stages:
#1. remove any old logs which contain edits older than any outstanding/unflushed edits in a memstore.
#2. if at this point we are still over the threshold, flush regions containing edits from the oldest log.
For the case you mention: "All edits in the oldest log are for regions that have moved to other servers" -- wouldn't stage #1 itself have removed the said log? At the end of stage #1, any logs remaining should at least have one memstore that's preventing that log from being reclaimed in stage #1, correct?
> NPE when trying to roll logs
> ----------------------------
>
> Key: HBASE-3239
> URL: https://issues.apache.org/jira/browse/HBASE-3239
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0
> Reporter: Prakash Khemani
> Assignee: Kannan Muthukkaruppan
> Priority: Blocker
> Fix For: 0.90.0
>
>
> Note from Kannan
> findMemstoresWithEditsEqualOrOlderThan() can return NULL it seems like. And we don't check NULL, before "region.length".
> regions = findMemstoresWithEditsEqualOrOlderThan(this.outputfiles.firstKey(),
> this.lastSeqWritten);
> StringBuilder sb = new StringBuilder();
> for (int i = 0; i < regions.length; i++) {
> ===
> Stack Trace
> 2010-11-15 19:19:54,258 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.1 GB, free=1.71 GB, max=7.81 GB, blocks=385740, accesses=7020255, hits=6329399, hitRatio=90.15%%, cachingAccesses=6765050, cachingHits=6329399, cachingHitsRatio=93.56%%, evictions=1, evicted=49911, evictedPerRun=49911.0
> 2010-11-15 19:21:05,204 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2010-11-15 19:21:05,211 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877154987, entries=649004, filesize=255069060. New hlog /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877665062
> 2010-11-15 19:21:05,222 ERROR org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,226 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=pumahbase042.snc5.facebook.com,60020,1289856892583, load=(requests=3476, regions=40, usedHeap=8388, maxHeap=15987): Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=1264.5834, regions=40, stores=70, storefiles=98, storefileIndexSize=35, memstoreSize=83, compactionQueueSize=0, usedHeap=8370, maxHeap=15987, blockCacheSize=6593768536, blockCacheFree=1788154792, blockCacheCount=388283, blockCacheHitRatio=90, blockCacheHitCachingRatio=93
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Log rolling failed
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
> 2010-11-15 19:21:07,255 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> ===
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3239) NPE when trying to roll logs
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933255#action_12933255 ]
Jean-Daniel Cryans commented on HBASE-3239:
-------------------------------------------
So we confirmed that the cluster only had HBASE-3198 and NOT HBASE-3208, which fixed this particular NPE.
@Kannan, would you mind rescoping the title of this jira to something more around handling the NPE rather than the NPE itself?
> NPE when trying to roll logs
> ----------------------------
>
> Key: HBASE-3239
> URL: https://issues.apache.org/jira/browse/HBASE-3239
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0
> Reporter: Prakash Khemani
> Assignee: Kannan Muthukkaruppan
> Priority: Blocker
> Fix For: 0.90.0
>
>
> Note from Kannan
> findMemstoresWithEditsEqualOrOlderThan() can return NULL it seems like. And we don't check NULL, before "region.length".
> regions = findMemstoresWithEditsEqualOrOlderThan(this.outputfiles.firstKey(),
> this.lastSeqWritten);
> StringBuilder sb = new StringBuilder();
> for (int i = 0; i < regions.length; i++) {
> ===
> Stack Trace
> 2010-11-15 19:19:54,258 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.1 GB, free=1.71 GB, max=7.81 GB, blocks=385740, accesses=7020255, hits=6329399, hitRatio=90.15%%, cachingAccesses=6765050, cachingHits=6329399, cachingHitsRatio=93.56%%, evictions=1, evicted=49911, evictedPerRun=49911.0
> 2010-11-15 19:21:05,204 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2010-11-15 19:21:05,211 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877154987, entries=649004, filesize=255069060. New hlog /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877665062
> 2010-11-15 19:21:05,222 ERROR org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,226 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=pumahbase042.snc5.facebook.com,60020,1289856892583, load=(requests=3476, regions=40, usedHeap=8388, maxHeap=15987): Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=1264.5834, regions=40, stores=70, storefiles=98, storefileIndexSize=35, memstoreSize=83, compactionQueueSize=0, usedHeap=8370, maxHeap=15987, blockCacheSize=6593768536, blockCacheFree=1788154792, blockCacheCount=388283, blockCacheHitRatio=90, blockCacheHitCachingRatio=93
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Log rolling failed
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
> 2010-11-15 19:21:07,255 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> ===
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3239) NPE when trying to roll logs
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933041#action_12933041 ]
Jean-Daniel Cryans commented on HBASE-3239:
-------------------------------------------
I think a possible reason why we haven't seen this before is HBASE-3198 & HBASE-3208, since we were archiving the logs prematurely.
Also regarding:
bq. All edits in the oldest log are for regions that have moved to other servers
I think it could also happen with splits.
> NPE when trying to roll logs
> ----------------------------
>
> Key: HBASE-3239
> URL: https://issues.apache.org/jira/browse/HBASE-3239
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0
> Reporter: Prakash Khemani
> Assignee: Kannan Muthukkaruppan
> Priority: Blocker
> Fix For: 0.90.0
>
>
> Note from Kannan
> findMemstoresWithEditsEqualOrOlderThan() can return NULL it seems like. And we don't check NULL, before "region.length".
> regions = findMemstoresWithEditsEqualOrOlderThan(this.outputfiles.firstKey(),
> this.lastSeqWritten);
> StringBuilder sb = new StringBuilder();
> for (int i = 0; i < regions.length; i++) {
> ===
> Stack Trace
> 2010-11-15 19:19:54,258 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=6.1 GB, free=1.71 GB, max=7.81 GB, blocks=385740, accesses=7020255, hits=6329399, hitRatio=90.15%%, cachingAccesses=6765050, cachingHits=6329399, cachingHitsRatio=93.56%%, evictions=1, evicted=49911, evictedPerRun=49911.0
> 2010-11-15 19:21:05,204 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs -- HDFS-200
> 2010-11-15 19:21:05,211 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877154987, entries=649004, filesize=255069060. New hlog /PUMAHBASE001-SNC5-HBASE/.logs/pumahbase042.snc5.facebook.com,60020,1289856892583/10.38.28.57%3A60020.1289877665062
> 2010-11-15 19:21:05,222 ERROR org.apache.hadoop.hbase.regionserver.LogRoller: Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,226 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server serverName=pumahbase042.snc5.facebook.com,60020,1289856892583, load=(requests=3476, regions=40, usedHeap=8388, maxHeap=15987): Log rolling failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.regionserver.wal.HLog.cleanOldLogs(HLog.java:648)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:528)
> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: request=1264.5834, regions=40, stores=70, storefiles=98, storefileIndexSize=35, memstoreSize=83, compactionQueueSize=0, usedHeap=8370, maxHeap=15987, blockCacheSize=6593768536, blockCacheFree=1788154792, blockCacheCount=388283, blockCacheHitRatio=90, blockCacheHitCachingRatio=93
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Log rolling failed
> 2010-11-15 19:21:05,227 INFO org.apache.hadoop.hbase.regionserver.LogRoller: LogRoller exiting.
> 2010-11-15 19:21:07,255 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
> ===
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.