You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Vladimir Rodionov <vr...@carrieriq.com> on 2013/07/11 20:10:53 UTC
Master server abort
This is happening in one of our small QA cluster.
HBase 0.94.6.1 (CDH 4.3.0)
1 master + 5 RS. Zk quorum is 1 (on master node)
We can not start the cluster:
In a log file I find some ERROR's and FATALs . FATAL's come first followed by ERRORs (this is important):
FATALs:
2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the region SMALL_GOLDENROD_2012-IDPROFILES,31,1363783108271.77a4640bfaecc907e0ea3535a16c56a8. that was online on sjc1-eng-qa04.carrieriq.com,60020,1373485278882
2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the region TEST_MM5550_INDEX-UPLOADS,9C,1363689771995.2ee2e6b81ee44ff790abf38275698d45. that was online on sjc1-eng-qa03.carrieriq.com,60020,1373485278616
2013-07-10 19:42:00,376 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the region SMALL_GOLDENROD_2012-IDPROFILES,F8,1363783108280.a0b1b6d003df84ca1404af942bcc9fbc. that was online on sjc1-eng-qa02.carrieriq.com,60020,1373485278611
2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the region TEST_MM5550_INDEX-UPLOADS,E0,1363689771998.36db41b10c86ac537542104c87950709. that was online on sjc1-eng-qa06.carrieriq.com,60020,1373485278668
2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the region SMALL_GOLDENROD_2012-IDPROFILES,FD,1363783108280.95ac4fb83f1bc2753aca0f6a914f6ff2. that was online on sjc1-eng-qa02.carrieriq.com,60020,1373485278611
2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the region TEST_MM5550_INDEX-UPLOADS,E5,1363689771999.65440a41f85b9dd70afd669280491363. that was online on sjc1-eng-qa06.carrieriq.com,60020,1373485278668
2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the region SMALL_GOLDENROD_2012-IDPROFILES,34,1363783108271.979f840723771588cf65910183ecf55c. that was online on sjc1-eng-qa04.carrieriq.com,60020,1373485278882
2013-07-10 19:42:00,376 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the region TEST_MM5550_INDEX-UPLOADS,A0,1363689771995.45f193f3dcfcfa76e705b5fa020e4309. that was online on sjc1-eng-qa03.carrieriq.com,60020,1373485278616
2013-07-10 19:42:00,377 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : packageindex,C0000000,1362756765100.7287ded900b6f6c14f22db5f9ae15d32. state=PENDING_OPEN, ts=1373485320376, server=sjc1-eng-qa06.carrieriq.com,60020,1373485278668 .. Cannot transit it to OFFLINE.
java.lang.IllegalStateException: Unexpected state : packageindex,C0000000,1362756765100.7287ded900b6f6c14f22db5f9ae15d32. state=PENDING_OPEN, ts=1373485320376, server=sjc1-eng-qa06.carrieriq.com,60020,1373485278668 .. Cannot transit it to OFFLINE.
at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1820)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1659)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2013-07-10 19:42:00,377 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the region XXX-IDPROFILES,36,1363783108271.65e59eae29b9d61d294d2dfcc3f08bd9. that was online on sjc1-eng-qa04.carrieriq.com,60020,1373485278882
2013-07-10 19:42:00,379 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
All errors in a log files are logged after the master started abort operation.
There are two types of ERRORs:
Type 1:
2013-07-10 19:42:01,084 ERROR org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: The znode of region TRIAL-UPLOADS,1C,1360234840368.a6944572c8e4a2447d3dcaa1d5029500. could not be deleted.
Type 2:
2013-07-10 19:42:01,085 ERROR org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit [OpenedRegionHandler-sjc1-eng-qa01.carrieriq.com,60000,1373485277977-8205] because the executor is missing. Is this process shutting down?
But again, I think that these errors are results of abort operation.
So, the question is what can the reason of a master sudden shutdown and where should I look for clues?
WARNINGs preceding first FATAL:
2013-07-10 19:41:18,196 WARN org.apache.hadoop.conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
2013-07-10 19:41:18,385 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/backup-masters/sjc1-eng-qa01.carrieriq.com,60000,1373485277977 already deleted, and this is not a retry
2013-07-10 19:41:18,942 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS
2013-07-10 19:41:19,208 WARN org.apache.hadoop.conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2013-07-10 19:41:19,494 WARN org.apache.hadoop.hbase.master.snapshot.SnapshotManager: Couldn't delete working snapshot directory: hdfs://sjc1-eng-qa01.carrieriq.com:9000/hbase/.snapshot/.tmp
2013-07-10 19:41:20,639 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS
2013-07-10 19:41:20,639 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS
2013-07-10 19:41:20,639 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS
2013-07-10 19:41:27,787 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/root-region-server already deleted, and this is not a retry
2013-07-10 19:41:27,818 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/-ROOT- already deleted, and this is not a retry
2013-07-10 19:41:29,562 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/.META. already deleted, and this is not a retry
2013-07-10 19:41:32,846 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/BATCH_OUTPUT_68_TEST-DIMENSIONS-1359838920000 already deleted, and this is not a retry
2013-07-10 19:41:32,909 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/BATCH_OUTPUT_68_TEST-DIMENSIONS-1360281600000 already deleted, and this is not a retry
2013-07-10 19:41:32,966 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/BATCH_OUTPUT_68_TS-DIMENSIONS-1360281540000 already deleted, and this is not a retry
2013-07-10 19:41:33,011 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/BATCH_SMOKE_98-DIMENSIONS-1358380800000 already deleted, and this is not a retry
2013-07-10 19:41:33,042 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/BATCH_SMOKE_TS-DIMENSIONS-1358121540000 already deleted, and this is not a retry
2013-07-10 19:41:33,070 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/BATCH_SYNTHETIC_UB-DIMENSIONS-1359590340000 already deleted, and this is not a retry
2013-07-10 19:41:33,107 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/BATCH_SYNTHETIC_UB-DIMENSIONS-1360281540000 already deleted, and this is not a retry
2013-07-10 19:41:33,128 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/CB_SMALL-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,184 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/CB_SMALL-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,226 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/COMPARISON_DB2_TS-DIMENSIONS-1327881540000 already deleted, and this is not a retry
2013-07-10 19:41:33,245 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/COMPARISON_DB2_TS-DIMENSIONS-1359503940000 already deleted, and this is not a retry
2013-07-10 19:41:33,264 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/DEVICERATIO_FILTER_111_TS-DIMENSIONS-1358467200000 already deleted, and this is not a retry
2013-07-10 19:41:33,288 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/DEVICE_LIST_TS-DIMENSIONS-1358467200000 already deleted, and this is not a retry
2013-07-10 19:41:33,306 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/DEVICE_RATIO_0_7_TS-DIMENSIONS-1358467200000 already deleted, and this is not a retry
2013-07-10 19:41:33,323 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/DEVICE_RATIO_109_TS-DIMENSIONS-1357603200000 already deleted, and this is not a retry
2013-07-10 19:41:33,338 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/DEVICE_RATIO_109_TS2-DIMENSIONS-1357603200000 already deleted, and this is not a retry
2013-07-10 19:41:33,352 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/DEVICE_RATIO_109_TS3-DIMENSIONS-1358467200000 already deleted, and this is not a retry
2013-07-10 19:41:33,364 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/GENERATED_UB-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,390 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/GENERATED_UB-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,416 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/GOLDENROD_PERF_INDEX-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,444 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/GOLDENROD_PERF_INDEX-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,467 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/INCOMING_UPLOADBINARIES_101229-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,491 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/INCOMING_UPLOADBINARIES_101229-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,514 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/INCOMING_UPLOADBINARIES_VE-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,535 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/INCOMING_UPLOADBINARIES_VE-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,557 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/INDEX_TEST3-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,578 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/INDEX_TEST3-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,596 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/IQADataStore already deleted, and this is not a retry
2013-07-10 19:41:33,603 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/MM_5550_TEST-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,619 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/MM_5550_TEST-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,634 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/MetaTable already deleted, and this is not a retry
2013-07-10 19:41:33,644 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/NO_FILTER_111_TS-DIMENSIONS-1358467200000 already deleted, and this is not a retry
2013-07-10 19:41:33,654 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/RTI_331_INDEX-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,669 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/RTI_331_INDEX-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,683 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/RTI_SEBY_TEST-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,702 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/RTI_SEBY_TEST-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,716 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/SMALL_GOLDENROD_2012-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,730 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/SMALL_GOLDENROD_2012-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,744 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/SYNTHETIC_UB-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,757 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/SYNTHETIC_UB-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,771 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/TEST_INDEX_TOOL-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,785 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/TEST_INDEX_TOOL-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,810 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/TEST_MM5550_INDEX-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,823 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/TEST_MM5550_INDEX-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,837 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/TMO_2012_DATASET-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,850 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/TMO_2012_DATASET-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,863 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/TRIAL-IDPROFILES already deleted, and this is not a retry
2013-07-10 19:41:33,876 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/TRIAL-UPLOADS already deleted, and this is not a retry
2013-07-10 19:41:33,889 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/table/packageindex already deleted, and this is not a retry
2013-07-10 19:41:56,349 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 61e36043936d733a370c8f6dddb83566 from server sjc1-eng-qa04.carrieriq.com,60020,1373485278882 but region was in the state INDEX_TEST3-IDPROFILES,16,1363283705765.61e36043936d733a370c8f6dddb83566. state=OPEN, ts=1373485312867, server=sjc1-eng-qa04.carrieriq.com,60020,1373485278882 and not in expected OFFLINE, PENDING_OPEN or OPENING states
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: Otis Gospodnetic [otis.gospodnetic@gmail.com]
Sent: Thursday, July 11, 2013 10:55 AM
To: user@hbase.apache.org
Subject: Re: G1 before/after GC time graph
Correct!
Otis
Solr & ElasticSearch Support
http://sematext.com/
On Jul 11, 2013 1:18 PM, "Asaf Mesika" <as...@gmail.com> wrote:
> This means you can safely run Hadoop and Hbase on jvm 7?
> We were just considering switching in production to java 7.
>
> On Thursday, July 11, 2013, Azuryy Yu wrote:
>
> > Otis,
> >
> > I will do this test, maybe on the end of this month. because I haven't
> big
> > memory server for test now util the end of this month.
> >
> >
> >
> >
> > On Thu, Jul 11, 2013 at 9:38 AM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Probably not any time in the near future. We could do it, plop SPM in
> > > it and watch, but don't have time for it now.
> > >
> > > If anyone wants to give it a shot and use SPM to capture various
> > > JVM/HBase/server metrics, let me know if help around SPM is needed,
> > > longer trials, or whatever.
> > >
> > > Otis
> > > --
> > > HBase Performance Monitoring -- http://sematext.com/spm
> > >
> > >
> > >
> > > On Wed, Jul 10, 2013 at 5:27 PM, Kevin O'dell <
> kevin.odell@cloudera.com>
> > > wrote:
> > > > Anyway you can test this with a 32GB heap? :)
> > > >
> > > >
> > > > On Wed, Jul 10, 2013 at 5:23 PM, Otis Gospodnetic <
> > > > otis.gospodnetic@gmail.com> wrote:
> > > >
> > > >> But, very importantly:
> > > >>
> > > >> CPU after changes: https://apps.sematext.com/spm/s/CpZ9aPhL9L
> > > >> Load after changes: https://apps.sematext.com/spm/s/i7f3x6MXdZ
> > > >>
> > > >> So in practical terms, use something like -XX:MaxGCPauseMillis=200
> > > >> -XX:NewRatio=4 to force short GC pauses, but count on having more of
> > > >> them and CPU sweating more. This gives you smaller pauses, no big
> STW
> > > >> pauses, but be prepared to pay in CPU cycles.
> > > >>
> > > >> HTH
> > > >>
> > > >> Otis
> > > >> --
> > > >> HBase Performance Monitoring -- http://sematext.com/spm
> > > >>
> > > >>
> > > >>
> > > >> On Wed, Jul 10, 2013 at 4:20 PM, Otis Gospodnetic
> > > >> <ot...@gmail.com> wrote:
> > > >> > Not big in this case - 1800m
> > > >> >
> > > >> > Otis
> > > >> > --
> > > >> > HBase Performance Monitoring -- http://sematext.com/spm
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Wed, Jul 10, 2013 at 3:57 PM, Vladimir Rodionov
> > > >> > <vr...@carrieriq.com> wrote:
> > > >> >>
> > > >> >> What is -Xmx heap size?
> > > >> >>
> > > >> >> Best regards,
> > > >> >> Vladimir Rodionov
> > > >> >> Principal Platform Engineer
> > > >> >> Carrier IQ, www.carrieriq.com
> > > >> >> e-mail: vrodionov@carrieriq.com
> > > >> >>
> > > >> >> ________________________________________
> > > >> >> From: Otis Gospodnetic [otis.gospodnetic@gmail.com]
> > > >> >> Sent: Wednesday, July 10, 2013 12:00 PM
> > > >> >> To: user@hbase.apache.org
> > > >> >> Subject: G1 before/after GC time graph
> > > >> >>
> > > >> >> Hi,
> > > >> >>
> > > >> >> There was talk of G1 and HBase here over the past few days. I
> > wanted
> > > >> >> to share a fresh and telling SPM graph showing G1 + some other
> > params
> > > >> >> in action:
> > > >> >>
> > > >> >> Result first: https://apps.sematext.com/spm/s/wN0hKVixxX
> > > >> >>
> > > >> >> This is what we had before:
> > > >> >> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseG1GC
> > > >> >> -XX:InitiatingHeapOccupancyPercent=35 -XX:G1ReservePercent=25
> > > >> >>
> > > >> >> And then we added:
> > > >> >>
>
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.
Re: Master server abort
Posted by Enis Söztutar <en...@gmail.com>.
I've seen a similar stack trace in some test as well, and opened the issue
https://issues.apache.org/jira/browse/HBASE-8912 for tracking this.
This looks like a problem in AssignmentManager that fails to recognize a
valid state transition, but I did not have the time to look into it
further. We'll spend some time to fix this issue, given that this affect
production deployments.
Can you please attach your logs at the issue as well.
Enis
On Thu, Jul 11, 2013 at 11:10 AM, Vladimir Rodionov <vrodionov@carrieriq.com
> wrote:
> This is happening in one of our small QA cluster.
> HBase 0.94.6.1 (CDH 4.3.0)
>
> 1 master + 5 RS. Zk quorum is 1 (on master node)
>
> We can not start the cluster:
>
> In a log file I find some ERROR's and FATALs . FATAL's come first followed
> by ERRORs (this is important):
>
> FATALs:
>
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> SMALL_GOLDENROD_2012-IDPROFILES,31,1363783108271.77a4640bfaecc907e0ea3535a16c56a8.
> that was online on sjc1-eng-qa04.carrieriq.com,60020,1373485278882
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> TEST_MM5550_INDEX-UPLOADS,9C,1363689771995.2ee2e6b81ee44ff790abf38275698d45.
> that was online on sjc1-eng-qa03.carrieriq.com,60020,1373485278616
> 2013-07-10 19:42:00,376 FATAL org.apache.hadoop.hbase.master.HMaster:
> Master server abort: loaded coprocessors are: []
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> SMALL_GOLDENROD_2012-IDPROFILES,F8,1363783108280.a0b1b6d003df84ca1404af942bcc9fbc.
> that was online on sjc1-eng-qa02.carrieriq.com,60020,1373485278611
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> TEST_MM5550_INDEX-UPLOADS,E0,1363689771998.36db41b10c86ac537542104c87950709.
> that was online on sjc1-eng-qa06.carrieriq.com,60020,1373485278668
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> SMALL_GOLDENROD_2012-IDPROFILES,FD,1363783108280.95ac4fb83f1bc2753aca0f6a914f6ff2.
> that was online on sjc1-eng-qa02.carrieriq.com,60020,1373485278611
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> TEST_MM5550_INDEX-UPLOADS,E5,1363689771999.65440a41f85b9dd70afd669280491363.
> that was online on sjc1-eng-qa06.carrieriq.com,60020,1373485278668
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> SMALL_GOLDENROD_2012-IDPROFILES,34,1363783108271.979f840723771588cf65910183ecf55c.
> that was online on sjc1-eng-qa04.carrieriq.com,60020,1373485278882
> 2013-07-10 19:42:00,376 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region
> TEST_MM5550_INDEX-UPLOADS,A0,1363689771995.45f193f3dcfcfa76e705b5fa020e4309.
> that was online on sjc1-eng-qa03.carrieriq.com,60020,1373485278616
> 2013-07-10 19:42:00,377 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unexpected state :
> packageindex,C0000000,1362756765100.7287ded900b6f6c14f22db5f9ae15d32.
> state=PENDING_OPEN, ts=1373485320376, server=sjc1-eng-qa06.carrieriq.com,60020,1373485278668
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> packageindex,C0000000,1362756765100.7287ded900b6f6c14f22db5f9ae15d32.
> state=PENDING_OPEN, ts=1373485320376, server=sjc1-eng-qa06.carrieriq.com,60020,1373485278668
> .. Cannot transit it to OFFLINE.
> at
> org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1820)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1659)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> at
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 2013-07-10 19:42:00,377 INFO
> org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the
> region XXX-IDPROFILES,36,1363783108271.65e59eae29b9d61d294d2dfcc3f08bd9.
> that was online on sjc1-eng-qa04.carrieriq.com,60020,1373485278882
> 2013-07-10 19:42:00,379 INFO org.apache.hadoop.hbase.master.HMaster:
> Aborting
>
>
> All errors in a log files are logged after the master started abort
> operation.
>
> There are two types of ERRORs:
>
> Type 1:
>
> 2013-07-10 19:42:01,084 ERROR
> org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: The znode of
> region TRIAL-UPLOADS,1C,1360234840368.a6944572c8e4a2447d3dcaa1d5029500.
> could not be deleted.
>
> Type 2:
>
> 2013-07-10 19:42:01,085 ERROR
> org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit [
> OpenedRegionHandler-sjc1-eng-qa01.carrieriq.com,60000,1373485277977-8205]
> because the executor is missing. Is this process shutting down?
>
>
> But again, I think that these errors are results of abort operation.
>
> So, the question is what can the reason of a master sudden shutdown and
> where should I look for clues?
>
> WARNINGs preceding first FATAL:
>
> 2013-07-10 19:41:18,196 WARN org.apache.hadoop.conf.Configuration:
> mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
> 2013-07-10 19:41:18,385 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/backup-masters/sjc1-eng-qa01.carrieriq.com,60000,1373485277977
> already deleted, and this is not a retry
> 2013-07-10 19:41:18,942 WARN org.apache.hadoop.conf.Configuration:
> fs.default.name is deprecated. Instead, use fs.defaultFS
> 2013-07-10 19:41:19,208 WARN org.apache.hadoop.conf.Configuration:
> hadoop.native.lib is deprecated. Instead, use io.native.lib.available
> 2013-07-10 19:41:19,494 WARN
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager: Couldn't delete
> working snapshot directory: hdfs://
> sjc1-eng-qa01.carrieriq.com:9000/hbase/.snapshot/.tmp
> 2013-07-10 19:41:20,639 WARN org.apache.hadoop.conf.Configuration:
> fs.default.name is deprecated. Instead, use fs.defaultFS
> 2013-07-10 19:41:20,639 WARN org.apache.hadoop.conf.Configuration:
> fs.default.name is deprecated. Instead, use fs.defaultFS
> 2013-07-10 19:41:20,639 WARN org.apache.hadoop.conf.Configuration:
> fs.default.name is deprecated. Instead, use fs.defaultFS
> 2013-07-10 19:41:27,787 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/root-region-server already deleted, and this is not a retry
> 2013-07-10 19:41:27,818 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/-ROOT- already deleted, and this is not a retry
> 2013-07-10 19:41:29,562 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/.META. already deleted, and this is not a retry
> 2013-07-10 19:41:32,846 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/BATCH_OUTPUT_68_TEST-DIMENSIONS-1359838920000 already deleted,
> and this is not a retry
> 2013-07-10 19:41:32,909 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/BATCH_OUTPUT_68_TEST-DIMENSIONS-1360281600000 already deleted,
> and this is not a retry
> 2013-07-10 19:41:32,966 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/BATCH_OUTPUT_68_TS-DIMENSIONS-1360281540000 already deleted,
> and this is not a retry
> 2013-07-10 19:41:33,011 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/BATCH_SMOKE_98-DIMENSIONS-1358380800000 already deleted, and
> this is not a retry
> 2013-07-10 19:41:33,042 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/BATCH_SMOKE_TS-DIMENSIONS-1358121540000 already deleted, and
> this is not a retry
> 2013-07-10 19:41:33,070 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/BATCH_SYNTHETIC_UB-DIMENSIONS-1359590340000 already deleted,
> and this is not a retry
> 2013-07-10 19:41:33,107 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/BATCH_SYNTHETIC_UB-DIMENSIONS-1360281540000 already deleted,
> and this is not a retry
> 2013-07-10 19:41:33,128 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/CB_SMALL-IDPROFILES already deleted, and this is not a retry
> 2013-07-10 19:41:33,184 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/CB_SMALL-UPLOADS already deleted, and this is not a retry
> 2013-07-10 19:41:33,226 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/COMPARISON_DB2_TS-DIMENSIONS-1327881540000 already deleted,
> and this is not a retry
> 2013-07-10 19:41:33,245 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/COMPARISON_DB2_TS-DIMENSIONS-1359503940000 already deleted,
> and this is not a retry
> 2013-07-10 19:41:33,264 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/DEVICERATIO_FILTER_111_TS-DIMENSIONS-1358467200000 already
> deleted, and this is not a retry
> 2013-07-10 19:41:33,288 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/DEVICE_LIST_TS-DIMENSIONS-1358467200000 already deleted, and
> this is not a retry
> 2013-07-10 19:41:33,306 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/DEVICE_RATIO_0_7_TS-DIMENSIONS-1358467200000 already deleted,
> and this is not a retry
> 2013-07-10 19:41:33,323 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/DEVICE_RATIO_109_TS-DIMENSIONS-1357603200000 already deleted,
> and this is not a retry
> 2013-07-10 19:41:33,338 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/DEVICE_RATIO_109_TS2-DIMENSIONS-1357603200000 already deleted,
> and this is not a retry
> 2013-07-10 19:41:33,352 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/DEVICE_RATIO_109_TS3-DIMENSIONS-1358467200000 already deleted,
> and this is not a retry
> 2013-07-10 19:41:33,364 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/GENERATED_UB-IDPROFILES already deleted, and this is not a
> retry
> 2013-07-10 19:41:33,390 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/GENERATED_UB-UPLOADS already deleted, and this is not a retry
> 2013-07-10 19:41:33,416 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/GOLDENROD_PERF_INDEX-IDPROFILES already deleted, and this is
> not a retry
> 2013-07-10 19:41:33,444 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/GOLDENROD_PERF_INDEX-UPLOADS already deleted, and this is not
> a retry
> 2013-07-10 19:41:33,467 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/INCOMING_UPLOADBINARIES_101229-IDPROFILES already deleted, and
> this is not a retry
> 2013-07-10 19:41:33,491 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/INCOMING_UPLOADBINARIES_101229-UPLOADS already deleted, and
> this is not a retry
> 2013-07-10 19:41:33,514 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/INCOMING_UPLOADBINARIES_VE-IDPROFILES already deleted, and
> this is not a retry
> 2013-07-10 19:41:33,535 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/INCOMING_UPLOADBINARIES_VE-UPLOADS already deleted, and this
> is not a retry
> 2013-07-10 19:41:33,557 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/INDEX_TEST3-IDPROFILES already deleted, and this is not a retry
> 2013-07-10 19:41:33,578 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/INDEX_TEST3-UPLOADS already deleted, and this is not a retry
> 2013-07-10 19:41:33,596 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/IQADataStore already deleted, and this is not a retry
> 2013-07-10 19:41:33,603 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/MM_5550_TEST-IDPROFILES already deleted, and this is not a
> retry
> 2013-07-10 19:41:33,619 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/MM_5550_TEST-UPLOADS already deleted, and this is not a retry
> 2013-07-10 19:41:33,634 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/MetaTable already deleted, and this is not a retry
> 2013-07-10 19:41:33,644 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/NO_FILTER_111_TS-DIMENSIONS-1358467200000 already deleted, and
> this is not a retry
> 2013-07-10 19:41:33,654 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/RTI_331_INDEX-IDPROFILES already deleted, and this is not a
> retry
> 2013-07-10 19:41:33,669 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/RTI_331_INDEX-UPLOADS already deleted, and this is not a retry
> 2013-07-10 19:41:33,683 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/RTI_SEBY_TEST-IDPROFILES already deleted, and this is not a
> retry
> 2013-07-10 19:41:33,702 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/RTI_SEBY_TEST-UPLOADS already deleted, and this is not a retry
> 2013-07-10 19:41:33,716 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/SMALL_GOLDENROD_2012-IDPROFILES already deleted, and this is
> not a retry
> 2013-07-10 19:41:33,730 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/SMALL_GOLDENROD_2012-UPLOADS already deleted, and this is not
> a retry
> 2013-07-10 19:41:33,744 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/SYNTHETIC_UB-IDPROFILES already deleted, and this is not a
> retry
> 2013-07-10 19:41:33,757 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/SYNTHETIC_UB-UPLOADS already deleted, and this is not a retry
> 2013-07-10 19:41:33,771 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/TEST_INDEX_TOOL-IDPROFILES already deleted, and this is not a
> retry
> 2013-07-10 19:41:33,785 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/TEST_INDEX_TOOL-UPLOADS already deleted, and this is not a
> retry
> 2013-07-10 19:41:33,810 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/TEST_MM5550_INDEX-IDPROFILES already deleted, and this is not
> a retry
> 2013-07-10 19:41:33,823 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/TEST_MM5550_INDEX-UPLOADS already deleted, and this is not a
> retry
> 2013-07-10 19:41:33,837 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/TMO_2012_DATASET-IDPROFILES already deleted, and this is not a
> retry
> 2013-07-10 19:41:33,850 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/TMO_2012_DATASET-UPLOADS already deleted, and this is not a
> retry
> 2013-07-10 19:41:33,863 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/TRIAL-IDPROFILES already deleted, and this is not a retry
> 2013-07-10 19:41:33,876 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/TRIAL-UPLOADS already deleted, and this is not a retry
> 2013-07-10 19:41:33,889 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/table/packageindex already deleted, and this is not a retry
> 2013-07-10 19:41:56,349 WARN
> org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for
> region 61e36043936d733a370c8f6dddb83566 from server
> sjc1-eng-qa04.carrieriq.com,60020,1373485278882 but region was in the
> state
> INDEX_TEST3-IDPROFILES,16,1363283705765.61e36043936d733a370c8f6dddb83566.
> state=OPEN, ts=1373485312867, server=sjc1-eng-qa04.carrieriq.com,60020,1373485278882
> and not in expected OFFLINE, PENDING_OPEN or OPENING states
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Otis Gospodnetic [otis.gospodnetic@gmail.com]
> Sent: Thursday, July 11, 2013 10:55 AM
> To: user@hbase.apache.org
> Subject: Re: G1 before/after GC time graph
>
> Correct!
>
> Otis
> Solr & ElasticSearch Support
> http://sematext.com/
> On Jul 11, 2013 1:18 PM, "Asaf Mesika" <as...@gmail.com> wrote:
>
> > This means you can safely run Hadoop and Hbase on jvm 7?
> > We were just considering switching in production to java 7.
> >
> > On Thursday, July 11, 2013, Azuryy Yu wrote:
> >
> > > Otis,
> > >
> > > I will do this test, maybe on the end of this month. because I haven't
> > big
> > > memory server for test now util the end of this month.
> > >
> > >
> > >
> > >
> > > On Thu, Jul 11, 2013 at 9:38 AM, Otis Gospodnetic <
> > > otis.gospodnetic@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > Probably not any time in the near future. We could do it, plop SPM
> in
> > > > it and watch, but don't have time for it now.
> > > >
> > > > If anyone wants to give it a shot and use SPM to capture various
> > > > JVM/HBase/server metrics, let me know if help around SPM is needed,
> > > > longer trials, or whatever.
> > > >
> > > > Otis
> > > > --
> > > > HBase Performance Monitoring -- http://sematext.com/spm
> > > >
> > > >
> > > >
> > > > On Wed, Jul 10, 2013 at 5:27 PM, Kevin O'dell <
> > kevin.odell@cloudera.com>
> > > > wrote:
> > > > > Anyway you can test this with a 32GB heap? :)
> > > > >
> > > > >
> > > > > On Wed, Jul 10, 2013 at 5:23 PM, Otis Gospodnetic <
> > > > > otis.gospodnetic@gmail.com> wrote:
> > > > >
> > > > >> But, very importantly:
> > > > >>
> > > > >> CPU after changes: https://apps.sematext.com/spm/s/CpZ9aPhL9L
> > > > >> Load after changes: https://apps.sematext.com/spm/s/i7f3x6MXdZ
> > > > >>
> > > > >> So in practical terms, use something like -XX:MaxGCPauseMillis=200
> > > > >> -XX:NewRatio=4 to force short GC pauses, but count on having more
> of
> > > > >> them and CPU sweating more. This gives you smaller pauses, no big
> > STW
> > > > >> pauses, but be prepared to pay in CPU cycles.
> > > > >>
> > > > >> HTH
> > > > >>
> > > > >> Otis
> > > > >> --
> > > > >> HBase Performance Monitoring -- http://sematext.com/spm
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Wed, Jul 10, 2013 at 4:20 PM, Otis Gospodnetic
> > > > >> <ot...@gmail.com> wrote:
> > > > >> > Not big in this case - 1800m
> > > > >> >
> > > > >> > Otis
> > > > >> > --
> > > > >> > HBase Performance Monitoring -- http://sematext.com/spm
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Jul 10, 2013 at 3:57 PM, Vladimir Rodionov
> > > > >> > <vr...@carrieriq.com> wrote:
> > > > >> >>
> > > > >> >> What is -Xmx heap size?
> > > > >> >>
> > > > >> >> Best regards,
> > > > >> >> Vladimir Rodionov
> > > > >> >> Principal Platform Engineer
> > > > >> >> Carrier IQ, www.carrieriq.com
> > > > >> >> e-mail: vrodionov@carrieriq.com
> > > > >> >>
> > > > >> >> ________________________________________
> > > > >> >> From: Otis Gospodnetic [otis.gospodnetic@gmail.com]
> > > > >> >> Sent: Wednesday, July 10, 2013 12:00 PM
> > > > >> >> To: user@hbase.apache.org
> > > > >> >> Subject: G1 before/after GC time graph
> > > > >> >>
> > > > >> >> Hi,
> > > > >> >>
> > > > >> >> There was talk of G1 and HBase here over the past few days. I
> > > wanted
> > > > >> >> to share a fresh and telling SPM graph showing G1 + some other
> > > params
> > > > >> >> in action:
> > > > >> >>
> > > > >> >> Result first: https://apps.sematext.com/spm/s/wN0hKVixxX
> > > > >> >>
> > > > >> >> This is what we had before:
> > > > >> >> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseG1GC
> > > > >> >> -XX:InitiatingHeapOccupancyPercent=35 -XX:G1ReservePercent=25
> > > > >> >>
> > > > >> >> And then we added:
> > > > >> >>
> >
>
> Confidentiality Notice: The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited. If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>