You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Abraham Tom <wo...@gmail.com> on 2015/11/11 05:09:51 UTC
hbase regions behaving badly
I am on CDH 5.4.8 hbase 1.0
I am migrating to a new cluster and the data is sourced from CDH 5.3 hbase
0.98.6
I took snapshots from the old cluster exported them to the new cluster - no
errors
cloned the table, flush, major_compact and balanced the regions
the configs are practically identical to the old cluster
hbck and fsck all check out with no inconsistencies found
yet when we try to load data via thrift the regions start splitting at an
outrageous pace.
Eventually the regions become offline starting a snowball affect. The
region exits are forcing the data to start transferring to other regions,
with that creates large GC activity and multiple regions in transition
before thrift can no longer execute any writes. This usually happens after
a few records. How we put is that we perform rowkey scans before puts to
ensure where we last left off.
Thrift eventually stops writing due to regions being offline and no longer
serving. The regions eventually stop splitting and all the GC eventually
complete. Once things settle back down, I am able to perform hbck and fsck
and it again comes back with no inconsistencies
I need ideas as to what is causing this
--
Abraham Tom
Email: work2much@gmail.com
Phone: 415-515-3621
Re: hbase regions behaving badly
Posted by Abraham Tom <wo...@gmail.com>.
HDFS health via fsck shows healthy
hbck shows no inconsistencies once all the regions finish transitioning
What I ended up doing is setting hbase.hregion.max.filesize to 4 GB from he
default CDH setting of 10GB. The old cluster had it at 10GB when I
snapshot it and exported it.
I did the normal flush compact after restarting. Then I let it load some
data which actually forced split files again. I stopped loading and I let
hbase do its transfers and splitting
Once the transfers were done, enabled blockcache to be true on all tables
and raised the hbase.hregion.max.filesize to 4.8 gb
Now it seems to be stable, at least its not frantically splitting and
resplitting regions since my new limit is higher than my old limit
The primary reason for the change is that S3 only allows for files less
than 5GB in size.
I am guessing that auto region splitting is literally the size you specify
and if you don't give it some growth room for random updates it will cause
files to be split which could snowball out to larger overhead activity
On Thu, Nov 12, 2015 at 6:53 AM, Ted Yu <yu...@gmail.com> wrote:
> bq. 2015-11-11 12:05:35,179 DEBUG
> org.apache.hadoop.hdfs.BlockReaderFactory: Closed potentially stale remote
> peer
>
> Have you checked hdfs health ?
>
> Cheers
>
> On Wed, Nov 11, 2015 at 10:10 AM, Abraham Tom <wo...@gmail.com> wrote:
>
>> Thanks
>>
>> I also restarted in debug mode
>> and found that hbase is renewing my lease to other nodes rather quickly
>> logs here
>> http://pastebin.com/4EgsXCDd
>>
>>
>> On Wed, Nov 11, 2015 at 6:20 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Please note that IncreasingToUpperBoundRegionSplitPolicy was in effect.
>>>
>>> See http://hbase.apache.org/book.html#_custom_split_policies for
>>> related information (there is link to
>>> http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.html
>>> ).
>>>
>>> In the future, please consider using pastebin etc. for server log so
>>> that the message is concise.
>>>
>>> Cheers
>>>
>>> On Tue, Nov 10, 2015 at 10:17 PM, Abraham Tom <wo...@gmail.com>
>>> wrote:
>>>
>>>> my hbase-site.xml snippet
>>>>
>>>>
>>>> <property><name>hbase.mob.sweep.tool.compaction.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
>>>>
>>>> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
>>>>
>>>> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value><source>hbase-default.xml</source></property>
>>>>
>>>> <property><name>hbase.mob.sweep.tool.compaction.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
>>>>
>>>> <property><name>dfs.blockreport.split.threshold</name><value>1000000</value><source>hdfs-default.xml</source></property>
>>>>
>>>> <property><name>hbase.regionserver.regionSplitLimit</name><value>1000</value><source>hbase-default.xml</source></property>
>>>>
>>>> <property><name>hbase.splitlog.manager.timeout</name><value>120000</value><source>hbase-site.xml</source></property>
>>>>
>>>> <property><name>hbase.regionserver.region.split.policy</name><value>org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy</value><source>hbase-default.xml</source></property>
>>>>
>>>> <property><name>mapreduce.job.split.metainfo.maxsize</name><value>10000000</value><source>mapred-default.xml</source></property>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> and some master logs
>>>> tail -100
>>>> hbase-cmf-hbase-MASTER-hadoop4-private.wdc01.infra.ripple.com.log.out
>>>> 2015-11-10 22:31:16,205 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {5097e5a15891c375eca53b992584c52e state=PENDING_OPEN, ts=1447216273284,
>>>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>>> {5097e5a15891c375eca53b992584c52e state=OPENING, ts=1447216276205, server=
>>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>>> 2015-11-10 22:31:16,206 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> d3b39413ee1bc88a100cd96564c4c420 from
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>>>> 2015-11-10 22:31:16,206 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {cff8a38c23c3987fbbebec615f45f4ac state=OPENING, ts=1447216276185, server=
>>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>>> {cff8a38c23c3987fbbebec615f45f4ac state=OPEN, ts=1447216276206, server=
>>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>>> 2015-11-10 22:31:16,206 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> bc842e97c8ff7b0f3498b38510a9c1c1 from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,206 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {c73147aef08804451c26c1f8099008f2 state=PENDING_OPEN, ts=1447216273284,
>>>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>>> {c73147aef08804451c26c1f8099008f2 state=OPENING, ts=1447216276206, server=
>>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>>> 2015-11-10 22:31:16,207 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> cff8a38c23c3987fbbebec615f45f4ac from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,208 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {8a671fb436fb0d186b37211d847aa897 state=OPENING, ts=1447216276188, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {8a671fb436fb0d186b37211d847aa897 state=OPEN, ts=1447216276208, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,209 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> 8a671fb436fb0d186b37211d847aa897 from
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>>>> 2015-11-10 22:31:16,209 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {e920fab780cdd6ba183bc50a8b5d8c72 state=PENDING_OPEN, ts=1447216273276,
>>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING, ts=1447216276209, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,210 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {1258fdeaf4f00989a8282e5f9c215d7e state=OPENING, ts=1447216276193, server=
>>>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189} to
>>>> {1258fdeaf4f00989a8282e5f9c215d7e state=OPEN, ts=1447216276210, server=
>>>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
>>>> 2015-11-10 22:31:16,211 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> 1258fdeaf4f00989a8282e5f9c215d7e from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,213 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {02666182c37df5043acdcaca31ceb095 state=OPENING, ts=1447216276184, server=
>>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>>> {02666182c37df5043acdcaca31ceb095 state=OPEN, ts=1447216276213, server=
>>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>>> 2015-11-10 22:31:16,214 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=PENDING_OPEN, ts=1447216273284,
>>>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING, ts=1447216276214, server=
>>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>>> 2015-11-10 22:31:16,215 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> 02666182c37df5043acdcaca31ceb095 from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,216 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {4c694bcbe9679747e738a630aec33c3c state=OPENING, ts=1447216276200, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {4c694bcbe9679747e738a630aec33c3c state=OPEN, ts=1447216276216, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,217 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> 4c694bcbe9679747e738a630aec33c3c from
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
>>>> 2015-11-10 22:31:16,217 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {dc3169f842a4d19014f040c887b9acff state=PENDING_OPEN, ts=1447216273280,
>>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {dc3169f842a4d19014f040c887b9acff state=OPENING, ts=1447216276217, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,219 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {8320a058823353a19f69a10da6827953 state=OPENING, ts=1447216276198, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {8320a058823353a19f69a10da6827953 state=OPEN, ts=1447216276219, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,219 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> 8320a058823353a19f69a10da6827953 from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,220 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {edfd9220fe6c2a37fe7521101d490941 state=PENDING_OPEN, ts=1447216273142,
>>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {edfd9220fe6c2a37fe7521101d490941 state=OPENING, ts=1447216276220, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,220 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {1937fd49e94d42fe072a29ff7af885bb state=OPENING, ts=1447216276205, server=
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>>> {1937fd49e94d42fe072a29ff7af885bb state=OPEN, ts=1447216276220, server=
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>>> 2015-11-10 22:31:16,220 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {e4b861ca490c05fa4276e73e629b471a state=OPENING, ts=1447216276204, server=
>>>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189} to
>>>> {e4b861ca490c05fa4276e73e629b471a state=OPEN, ts=1447216276220, server=
>>>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
>>>> 2015-11-10 22:31:16,221 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> 1937fd49e94d42fe072a29ff7af885bb from
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
>>>> 2015-11-10 22:31:16,221 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {9ecf375c160ffe4616b8c4fb45ea8fba state=PENDING_OPEN, ts=1447216273274,
>>>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>>> {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING, ts=1447216276221, server=
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>>> 2015-11-10 22:31:16,221 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> e4b861ca490c05fa4276e73e629b471a from
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>>>> 2015-11-10 22:31:16,222 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {2c7572eeff0a72eceec10e0301d344a7 state=OPENING, ts=1447216276205, server=
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>>> {2c7572eeff0a72eceec10e0301d344a7 state=OPEN, ts=1447216276222, server=
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>>> 2015-11-10 22:31:16,222 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> 2c7572eeff0a72eceec10e0301d344a7 from
>>>> hadoop3-private.wdc01.infra.ripple.com,60020,1447208454196
>>>> 2015-11-10 22:31:16,223 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {cb7e639dbb58b1e47c1da538165c503c state=PENDING_OPEN, ts=1447216273288,
>>>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>>> {cb7e639dbb58b1e47c1da538165c503c state=OPENING, ts=1447216276223, server=
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>>> 2015-11-10 22:31:16,225 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {5097e5a15891c375eca53b992584c52e state=OPENING, ts=1447216276205, server=
>>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>>> {5097e5a15891c375eca53b992584c52e state=OPEN, ts=1447216276225, server=
>>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>>> 2015-11-10 22:31:16,225 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {c73147aef08804451c26c1f8099008f2 state=OPENING, ts=1447216276206, server=
>>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>>> {c73147aef08804451c26c1f8099008f2 state=OPEN, ts=1447216276225, server=
>>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>>> 2015-11-10 22:31:16,226 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {abddf225d895bd30e11205390fedc03e state=OPENING, ts=1447216276205, server=
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>>> {abddf225d895bd30e11205390fedc03e state=OPEN, ts=1447216276226, server=
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>>> 2015-11-10 22:31:16,226 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> 5097e5a15891c375eca53b992584c52e from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,226 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> c73147aef08804451c26c1f8099008f2 from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,227 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> abddf225d895bd30e11205390fedc03e from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,227 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {a5e6796ae343481b92b0180d817c1e17 state=PENDING_OPEN, ts=1447216273288,
>>>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>>> {a5e6796ae343481b92b0180d817c1e17 state=OPENING, ts=1447216276227, server=
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>>> 2015-11-10 22:31:16,229 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING, ts=1447216276214, server=
>>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPEN, ts=1447216276229, server=
>>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>>> 2015-11-10 22:31:16,230 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING, ts=1447216276209, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {e920fab780cdd6ba183bc50a8b5d8c72 state=OPEN, ts=1447216276230, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,230 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> dae9fb9b19a8d2076fd4be2b95aefe45 from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,230 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> e920fab780cdd6ba183bc50a8b5d8c72 from
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>>>> 2015-11-10 22:31:16,230 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {2c04351380cc266df609ab53ef0e56ad state=PENDING_OPEN, ts=1447216273276,
>>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {2c04351380cc266df609ab53ef0e56ad state=OPENING, ts=1447216276230, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,236 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {edfd9220fe6c2a37fe7521101d490941 state=OPENING, ts=1447216276220, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {edfd9220fe6c2a37fe7521101d490941 state=OPEN, ts=1447216276236, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,236 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING, ts=1447216276221, server=
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>>> {9ecf375c160ffe4616b8c4fb45ea8fba state=OPEN, ts=1447216276236, server=
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>>> 2015-11-10 22:31:16,237 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {dc3169f842a4d19014f040c887b9acff state=OPENING, ts=1447216276217, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {dc3169f842a4d19014f040c887b9acff state=OPEN, ts=1447216276237, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,240 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> edfd9220fe6c2a37fe7521101d490941 from
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
>>>> 2015-11-10 22:31:16,240 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=PENDING_OPEN, ts=1447216273280,
>>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING, ts=1447216276240, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,241 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {c2a9f77791512281e4491e0797e824e8 state=PENDING_OPEN, ts=1447216273277,
>>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {c2a9f77791512281e4491e0797e824e8 state=OPENING, ts=1447216276241, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,241 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> dc3169f842a4d19014f040c887b9acff from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,241 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> 9ecf375c160ffe4616b8c4fb45ea8fba from
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>>>> 2015-11-10 22:31:16,244 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {cb7e639dbb58b1e47c1da538165c503c state=OPENING, ts=1447216276223, server=
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>>> {cb7e639dbb58b1e47c1da538165c503c state=OPEN, ts=1447216276244, server=
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>>> 2015-11-10 22:31:16,245 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> cb7e639dbb58b1e47c1da538165c503c from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,246 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {a5e6796ae343481b92b0180d817c1e17 state=OPENING, ts=1447216276227, server=
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>>> {a5e6796ae343481b92b0180d817c1e17 state=OPEN, ts=1447216276246, server=
>>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>>> 2015-11-10 22:31:16,246 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> a5e6796ae343481b92b0180d817c1e17 from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,252 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {2c04351380cc266df609ab53ef0e56ad state=OPENING, ts=1447216276230, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {2c04351380cc266df609ab53ef0e56ad state=OPEN, ts=1447216276252, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,253 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> 2c04351380cc266df609ab53ef0e56ad from
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>>>> 2015-11-10 22:31:16,253 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {15701b8131ca505f6cce148d49cfbeee state=PENDING_OPEN, ts=1447216273280,
>>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {15701b8131ca505f6cce148d49cfbeee state=OPENING, ts=1447216276253, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,257 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING, ts=1447216276240, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPEN, ts=1447216276257, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,257 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {c2a9f77791512281e4491e0797e824e8 state=OPENING, ts=1447216276241, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {c2a9f77791512281e4491e0797e824e8 state=OPEN, ts=1447216276257, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,258 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> a4f70c6c01f31f6d3d65e0cf55200c80 from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,258 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> c2a9f77791512281e4491e0797e824e8 from
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>>>> 2015-11-10 22:31:16,258 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {7ce823a4e0f56bf8c854175409bb17b6 state=PENDING_OPEN, ts=1447216273280,
>>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING, ts=1447216276258, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,258 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {1fbbd37bdff125be32d352c98dffd2b1 state=PENDING_OPEN, ts=1447216273280,
>>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING, ts=1447216276258, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,280 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING, ts=1447216276258, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {1fbbd37bdff125be32d352c98dffd2b1 state=OPEN, ts=1447216276280, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,280 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING, ts=1447216276258, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {7ce823a4e0f56bf8c854175409bb17b6 state=OPEN, ts=1447216276280, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,281 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> 1fbbd37bdff125be32d352c98dffd2b1 from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,281 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> 7ce823a4e0f56bf8c854175409bb17b6 from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,281 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {cc52e0ba23708df10d696d0e33d9d137 state=PENDING_OPEN, ts=1447216273280,
>>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {cc52e0ba23708df10d696d0e33d9d137 state=OPENING, ts=1447216276281, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,281 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {21b24baeaf26e12d02f79f243076850a state=PENDING_OPEN, ts=1447216273280,
>>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {21b24baeaf26e12d02f79f243076850a state=OPENING, ts=1447216276281, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,306 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {15701b8131ca505f6cce148d49cfbeee state=OPENING, ts=1447216276253, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {15701b8131ca505f6cce148d49cfbeee state=OPEN, ts=1447216276306, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,307 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> 15701b8131ca505f6cce148d49cfbeee from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,325 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {cc52e0ba23708df10d696d0e33d9d137 state=OPENING, ts=1447216276281, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {cc52e0ba23708df10d696d0e33d9d137 state=OPEN, ts=1447216276325, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,325 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>>> {21b24baeaf26e12d02f79f243076850a state=OPENING, ts=1447216276281, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>>> {21b24baeaf26e12d02f79f243076850a state=OPEN, ts=1447216276325, server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>>> 2015-11-10 22:31:16,326 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> cc52e0ba23708df10d696d0e33d9d137 from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:31:16,326 INFO
>>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>>> 21b24baeaf26e12d02f79f243076850a from
>>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>>> 2015-11-10 22:36:01,090 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 22:41:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 22:46:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=89, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 22:47:34,680 INFO
>>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>>>> 10.125.122.237 set balanceSwitch=false
>>>> 2015-11-10 22:47:41,845 INFO
>>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>>>> 10.125.122.237 set balanceSwitch=true
>>>> 2015-11-10 22:51:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 22:56:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=149, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 23:01:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=179, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 23:06:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=209, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 23:11:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=239, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 23:16:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=269, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 23:17:34,662 INFO
>>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>>>> 10.125.122.237 set balanceSwitch=false
>>>> 2015-11-10 23:17:38,707 INFO
>>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>>>> 10.125.122.237 set balanceSwitch=true
>>>> 2015-11-10 23:21:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=299, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 23:26:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=329, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 23:31:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=359, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 23:31:01,096 INFO org.apache.hadoop.hbase.mob.MobFileCache:
>>>> MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted
>>>> files: 0
>>>> 2015-11-10 23:36:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=389, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 23:41:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=419, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 23:46:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=449, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 23:47:34,668 INFO
>>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>>>> 10.125.122.237 set balanceSwitch=false
>>>> 2015-11-10 23:47:44,843 INFO
>>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>>>> 10.125.122.237 set balanceSwitch=true
>>>> 2015-11-10 23:51:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=479, evicted=0, evictedPerRun=0.0
>>>> 2015-11-10 23:56:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=509, evicted=0, evictedPerRun=0.0
>>>> 2015-11-11 00:01:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=539, evicted=0, evictedPerRun=0.0
>>>> 2015-11-11 00:06:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=569, evicted=0, evictedPerRun=0.0
>>>> 2015-11-11 00:11:01,089 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>>> cachingHitsRatio=0,evictions=599, evicted=0, evictedPerRun=0.0
>>>>
>>>> Region server logs
>>>> tail -100
>>>> hbase-cmf-hbase-REGIONSERVER-hadoop4-private.wdc01.infra.ripple.com.log.out
>>>> 2015-11-10 22:31:16,255 INFO
>>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>>> cacheDataCompressed=false, prefetchOnOpen=false
>>>> 2015-11-10 22:31:16,255 INFO
>>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>>> major jitter 0.500000, min locality to compact 0.000000
>>>> 2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>>>> Updated row prod_control,,1434077692105.a4f70c6c01f31f6d3d65e0cf55200c80.
>>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>>>> 2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>>>> Updated row
>>>> prod_agg_account_payments,20140325000000|rH48e8bf5kNWuFRBwgm5sCH133cd9zgV29,1447197624013.c2a9f77791512281e4491e0797e824e8.
>>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>>>> 2015-11-10 22:31:16,259 INFO
>>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>>> cacheDataCompressed=false, prefetchOnOpen=false
>>>> 2015-11-10 22:31:16,259 INFO
>>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>>> cacheDataCompressed=false, prefetchOnOpen=false
>>>> 2015-11-10 22:31:16,259 INFO
>>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>>> major jitter 0.500000, min locality to compact 0.000000
>>>> 2015-11-10 22:31:16,259 INFO
>>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>>> major jitter 0.500000, min locality to compact 0.000000
>>>> 2015-11-10 22:31:16,262 INFO
>>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>>> cacheDataCompressed=false, prefetchOnOpen=false
>>>> 2015-11-10 22:31:16,262 INFO
>>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>>> major jitter 0.500000, min locality to compact 0.000000
>>>> 2015-11-10 22:31:16,265 INFO
>>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>>> cacheDataCompressed=false, prefetchOnOpen=false
>>>> 2015-11-10 22:31:16,265 INFO
>>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>>> major jitter 0.500000, min locality to compact 0.000000
>>>> 2015-11-10 22:31:16,265 INFO
>>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>>> cacheDataCompressed=false, prefetchOnOpen=false
>>>> 2015-11-10 22:31:16,265 INFO
>>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>>> major jitter 0.500000, min locality to compact 0.000000
>>>> 2015-11-10 22:31:16,267 INFO
>>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>>> cacheDataCompressed=false, prefetchOnOpen=false
>>>> 2015-11-10 22:31:16,268 INFO
>>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>>> major jitter 0.500000, min locality to compact 0.000000
>>>> 2015-11-10 22:31:16,272 INFO
>>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>>> cacheDataCompressed=false, prefetchOnOpen=false
>>>> 2015-11-10 22:31:16,273 INFO
>>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>>> major jitter 0.500000, min locality to compact 0.000000
>>>> 2015-11-10 22:31:16,274 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>>>> 1fbbd37bdff125be32d352c98dffd2b1; next sequenceid=180083
>>>> 2015-11-10 22:31:16,274 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>>>> 7ce823a4e0f56bf8c854175409bb17b6; next sequenceid=21535
>>>> 2015-11-10 22:31:16,275 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>>>> for
>>>> prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1.
>>>> 2015-11-10 22:31:16,276 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>>>> for
>>>> prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6.
>>>> 2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>>>> Updated row
>>>> prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1.
>>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>>>> 2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>>>> Updated row
>>>> prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6.
>>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>>>> 2015-11-10 22:31:16,282 INFO
>>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>>> cacheDataCompressed=false, prefetchOnOpen=false
>>>> 2015-11-10 22:31:16,282 INFO
>>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>>> cacheDataCompressed=false, prefetchOnOpen=false
>>>> 2015-11-10 22:31:16,282 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>>>> 15701b8131ca505f6cce148d49cfbeee; next sequenceid=1029
>>>> 2015-11-10 22:31:16,283 INFO
>>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>>> major jitter 0.500000, min locality to compact 0.000000
>>>> 2015-11-10 22:31:16,283 INFO
>>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>>> major jitter 0.500000, min locality to compact 0.000000
>>>> 2015-11-10 22:31:16,284 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>>>> for
>>>> prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee.
>>>> 2015-11-10 22:31:16,304 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>>>> Updated row
>>>> prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee.
>>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>>>> 2015-11-10 22:31:16,308 INFO
>>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>>> cacheDataCompressed=false, prefetchOnOpen=false
>>>> 2015-11-10 22:31:16,308 INFO
>>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>>> major jitter 0.500000, min locality to compact 0.000000
>>>> 2015-11-10 22:31:16,308 INFO
>>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>>> cacheDataCompressed=false, prefetchOnOpen=false
>>>> 2015-11-10 22:31:16,309 INFO
>>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>>> major jitter 0.500000, min locality to compact 0.000000
>>>> 2015-11-10 22:31:16,318 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>>>> 21b24baeaf26e12d02f79f243076850a; next sequenceid=114628
>>>> 2015-11-10 22:31:16,318 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>>>> cc52e0ba23708df10d696d0e33d9d137; next sequenceid=161394
>>>> 2015-11-10 22:31:16,320 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>>>> for
>>>> prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a.
>>>> 2015-11-10 22:31:16,320 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>>>> for prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD|2014111212
>>>> ,1447198690089.cc52e0ba23708df10d696d0e33d9d137.
>>>> 2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>>>> Updated row prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD|
>>>> 2014111212,1447198690089.cc52e0ba23708df10d696d0e33d9d137. with server=
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>>>> 2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>>>> Updated row
>>>> prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a.
>>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>>>> 2015-11-10 22:31:21,184 WARN org.apache.hadoop.metrics2.lib.Interns:
>>>> Metrics intern cache overflow at 2011 for
>>>> RegionServer={RegionServer=MetricsInfoImpl{name=RegionServer,
>>>> description=RegionServer}, RegionServer
>>>> record=MetricsInfoImpl{name=RegionServer, description=RegionServer record}}
>>>> 2015-11-10 22:31:22,377 WARN
>>>> org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil:
>>>> hbase.regionserver.global.memstore.upperLimit is deprecated by
>>>> hbase.regionserver.global.memstore.size
>>>> 2015-11-10 22:31:22,403 INFO
>>>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3
>>>> (all) file(s) in f of
>>>> prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041.
>>>> into d98861e6b5364d819d11c3f3e1c8fb56(size=180.4 M), total size for store
>>>> is 180.4 M. This selection was in queue for 0sec, and took 7sec to execute.
>>>> 2015-11-10 22:31:22,405 INFO
>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>>>> compaction: Request =
>>>> regionName=prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041.,
>>>> storeName=f, fileCount=3, fileSize=180.4 M, priority=27,
>>>> time=741524299592168; duration=7sec
>>>> 2015-11-10 22:31:22,413 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on f in
>>>> region
>>>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
>>>> 2015-11-10 22:31:22,413 INFO
>>>> org.apache.hadoop.hbase.regionserver.HStore: Starting compaction of 3
>>>> file(s) in f of
>>>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
>>>> into tmpdir=hdfs://
>>>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_transactions/72f5a6c7dba1e486eae6b75f4e9f0d5d/.tmp,
>>>> totalSize=82.2 M
>>>> 2015-11-10 22:31:22,416 INFO
>>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>>> cacheDataCompressed=false, prefetchOnOpen=false
>>>> 2015-11-10 22:31:25,301 INFO
>>>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3
>>>> (all) file(s) in f of
>>>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
>>>> into f7678de8dd8149428938a0aedfcbcaf9(size=82.2 M), total size for store is
>>>> 82.2 M. This selection was in queue for 0sec, and took 2sec to execute.
>>>> 2015-11-10 22:31:25,302 INFO
>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>>>> compaction: Request =
>>>> regionName=prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.,
>>>> storeName=f, fileCount=3, fileSize=82.2 M, priority=27,
>>>> time=741532147583369; duration=2sec
>>>> 2015-11-10 22:31:25,305 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on f in
>>>> region
>>>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
>>>> 2015-11-10 22:31:25,305 INFO
>>>> org.apache.hadoop.hbase.regionserver.HStore: Starting compaction of 3
>>>> file(s) in f of
>>>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
>>>> into tmpdir=hdfs://
>>>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_offers/05284b692219df79852f3deaa2ce1927/.tmp,
>>>> totalSize=218.4 M
>>>> 2015-11-10 22:31:25,310 INFO
>>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>>> cacheDataCompressed=false, prefetchOnOpen=false
>>>> 2015-11-10 22:31:39,481 INFO
>>>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3
>>>> (all) file(s) in f of
>>>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
>>>> into 4aea6e4301f848ff9bf727f4e511978a(size=214.6 M), total size for store
>>>> is 214.6 M. This selection was in queue for 0sec, and took 14sec to execute.
>>>> 2015-11-10 22:31:39,481 INFO
>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>>>> compaction: Request =
>>>> regionName=prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.,
>>>> storeName=f, fileCount=3, fileSize=218.4 M, priority=27,
>>>> time=741535039619615; duration=14sec
>>>> 2015-11-10 22:31:39,484 INFO
>>>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on d in
>>>> region
>>>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
>>>> 2015-11-10 22:31:39,484 INFO
>>>> org.apache.hadoop.hbase.regionserver.HStore: Starting compaction of 3
>>>> file(s) in d of
>>>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
>>>> into tmpdir=hdfs://
>>>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_balance_changes/40d39088cc3efe2d230a7d6f8d7b205b/.tmp,
>>>> totalSize=197.7 M
>>>> 2015-11-10 22:31:39,490 INFO
>>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>>> cacheDataCompressed=false, prefetchOnOpen=false
>>>> 2015-11-10 22:31:52,393 INFO
>>>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3
>>>> (all) file(s) in d of
>>>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
>>>> into 1e48a57cdb544e898e78c9e8276ec05b(size=197.7 M), total size for store
>>>> is 197.7 M. This selection was in queue for 0sec, and took 12sec to execute.
>>>> 2015-11-10 22:31:52,393 INFO
>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>>>> compaction: Request =
>>>> regionName=prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.,
>>>> storeName=d, fileCount=3, fileSize=197.7 M, priority=27,
>>>> time=741549219351203; duration=12sec
>>>> 2015-11-10 22:32:01,402 INFO
>>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Atomically
>>>> moving hadoop3-private.wdc01.infra.ripple.com,60020,1447102836904's
>>>> wals to my queue
>>>> 2015-11-10 22:32:01,405 WARN
>>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer 1 didn't
>>>> exist, skipping the replay
>>>> 2015-11-10 22:32:01,406 WARN
>>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer
>>>> 1-hadoop4-private.wdc01.infra.ripple.com,60020,1447102248964 didn't
>>>> exist, skipping the replay
>>>> 2015-11-10 22:32:01,412 WARN
>>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Got exception
>>>> in copyQueuesFromRSUsingMulti:
>>>> org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode
>>>> = Directory not empty
>>>> at
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
>>>> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1152)
>>>> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:1118)
>>>> at
>>>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:685)
>>>> at
>>>> org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1550)
>>>> at
>>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.copyQueuesFromRSUsingMulti(ReplicationQueuesZKImpl.java:340)
>>>> at
>>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.claimQueues(ReplicationQueuesZKImpl.java:172)
>>>> at
>>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:570)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 2015-11-10 22:36:01,223 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=3.78 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=4, accesses=63330, hits=6,
>>>> hitRatio=0.01%, , cachingAccesses=21, cachingHits=4,
>>>> cachingHitsRatio=19.05%, evictions=29, evicted=3,
>>>> evictedPerRun=0.1034482792019844
>>>> 2015-11-10 22:41:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.15 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=7, accesses=63856, hits=8,
>>>> hitRatio=0.01%, , cachingAccesses=32, cachingHits=6,
>>>> cachingHitsRatio=18.75%, evictions=59, evicted=3,
>>>> evictedPerRun=0.050847455859184265
>>>> 2015-11-10 22:46:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
>>>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
>>>> cachingHitsRatio=21.43%, evictions=89, evicted=3,
>>>> evictedPerRun=0.033707864582538605
>>>> 2015-11-10 22:51:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
>>>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
>>>> cachingHitsRatio=21.43%, evictions=119, evicted=3,
>>>> evictedPerRun=0.02521008439362049
>>>> 2015-11-10 22:56:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
>>>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
>>>> cachingHitsRatio=21.43%, evictions=149, evicted=3,
>>>> evictedPerRun=0.020134227350354195
>>>> 2015-11-10 23:01:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63872, hits=13,
>>>> hitRatio=0.02%, , cachingAccesses=48, cachingHits=11,
>>>> cachingHitsRatio=22.92%, evictions=179, evicted=3,
>>>> evictedPerRun=0.016759777441620827
>>>> 2015-11-10 23:06:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63890, hits=21,
>>>> hitRatio=0.03%, , cachingAccesses=66, cachingHits=19,
>>>> cachingHitsRatio=28.79%, evictions=209, evicted=3,
>>>> evictedPerRun=0.014354066923260689
>>>> 2015-11-10 23:11:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64416, hits=26,
>>>> hitRatio=0.04%, , cachingAccesses=77, cachingHits=24,
>>>> cachingHitsRatio=31.17%, evictions=239, evicted=3,
>>>> evictedPerRun=0.012552301399409771
>>>> 2015-11-10 23:16:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
>>>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
>>>> cachingHitsRatio=33.33%, evictions=269, evicted=3,
>>>> evictedPerRun=0.011152416467666626
>>>> 2015-11-10 23:21:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
>>>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
>>>> cachingHitsRatio=33.33%, evictions=299, evicted=3,
>>>> evictedPerRun=0.010033444501459599
>>>> 2015-11-10 23:26:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
>>>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
>>>> cachingHitsRatio=33.33%, evictions=329, evicted=3,
>>>> evictedPerRun=0.009118541143834591
>>>> 2015-11-10 23:31:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64432, hits=34,
>>>> hitRatio=0.05%, , cachingAccesses=93, cachingHits=32,
>>>> cachingHitsRatio=34.41%, evictions=359, evicted=3,
>>>> evictedPerRun=0.008356546051800251
>>>> 2015-11-10 23:31:01,228 INFO org.apache.hadoop.hbase.mob.MobFileCache:
>>>> MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted
>>>> files: 0
>>>> 2015-11-10 23:31:06,790 INFO
>>>> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Rolled WAL /hbase/WALs/
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/
>>>> hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
>>>> with entries=247, filesize=125.29 KB; new WAL /hbase/WALs/
>>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/
>>>> hadoop4-private.wdc01.infra.ripple.com
>>>> %2C60020%2C1447216261115.null0.1447219866745
>>>> 2015-11-10 23:31:06,792 INFO
>>>> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Archiving hdfs://
>>>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/WALs/hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
>>>> to hdfs://
>>>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/oldWALs/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
>>>> 2015-11-10 23:36:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64450, hits=42,
>>>> hitRatio=0.07%, , cachingAccesses=111, cachingHits=40,
>>>> cachingHitsRatio=36.04%, evictions=389, evicted=3,
>>>> evictedPerRun=0.007712082471698523
>>>> 2015-11-10 23:41:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64976, hits=47,
>>>> hitRatio=0.07%, , cachingAccesses=122, cachingHits=45,
>>>> cachingHitsRatio=36.89%, evictions=419, evicted=3,
>>>> evictedPerRun=0.0071599045768380165
>>>> 2015-11-10 23:46:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
>>>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
>>>> cachingHitsRatio=37.88%, evictions=449, evicted=3,
>>>> evictedPerRun=0.00668151443824172
>>>> 2015-11-10 23:51:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
>>>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
>>>> cachingHitsRatio=37.88%, evictions=479, evicted=3,
>>>> evictedPerRun=0.00626304792240262
>>>> 2015-11-10 23:56:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
>>>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
>>>> cachingHitsRatio=37.88%, evictions=509, evicted=3,
>>>> evictedPerRun=0.005893909838050604
>>>> 2015-11-11 00:01:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64992, hits=55,
>>>> hitRatio=0.08%, , cachingAccesses=138, cachingHits=53,
>>>> cachingHitsRatio=38.41%, evictions=539, evicted=3,
>>>> evictedPerRun=0.005565862637013197
>>>> 2015-11-11 00:06:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65010, hits=63,
>>>> hitRatio=0.10%, , cachingAccesses=156, cachingHits=61,
>>>> cachingHitsRatio=39.10%, evictions=569, evicted=3,
>>>> evictedPerRun=0.005272407550364733
>>>> 2015-11-11 00:11:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65536, hits=68,
>>>> hitRatio=0.10%, , cachingAccesses=167, cachingHits=66,
>>>> cachingHitsRatio=39.52%, evictions=599, evicted=3,
>>>> evictedPerRun=0.00500834733247757
>>>> 2015-11-11 00:16:01,222 INFO
>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65546, hits=73,
>>>> hitRatio=0.11%, , cachingAccesses=177, cachingHits=71,
>>>> cachingHitsRatio=40.11%, evictions=629, evicted=3,
>>>> evictedPerRun=0.004769475199282169
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Nov 10, 2015 at 8:15 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> bq. The region exits are forcing the data to start transferring to
>>>>> other regions,
>>>>>
>>>>> Can you elaborate a bit more on the above ? Did you mean some regions
>>>>> were transferred to other region server(s) ?
>>>>>
>>>>> Can you determined the effective RegionSplitPolicy ?
>>>>>
>>>>> Snippets of server logs would help illustrate the situation.
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Tue, Nov 10, 2015 at 8:09 PM, Abraham Tom <wo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I am on CDH 5.4.8 hbase 1.0
>>>>>>
>>>>>> I am migrating to a new cluster and the data is sourced from CDH 5.3
>>>>>> hbase
>>>>>> 0.98.6
>>>>>>
>>>>>> I took snapshots from the old cluster exported them to the new
>>>>>> cluster - no
>>>>>> errors
>>>>>>
>>>>>> cloned the table, flush, major_compact and balanced the regions
>>>>>>
>>>>>> the configs are practically identical to the old cluster
>>>>>>
>>>>>>
>>>>>>
>>>>>> hbck and fsck all check out with no inconsistencies found
>>>>>>
>>>>>>
>>>>>>
>>>>>> yet when we try to load data via thrift the regions start splitting
>>>>>> at an
>>>>>> outrageous pace.
>>>>>>
>>>>>> Eventually the regions become offline starting a snowball affect. The
>>>>>> region exits are forcing the data to start transferring to other
>>>>>> regions,
>>>>>> with that creates large GC activity and multiple regions in transition
>>>>>> before thrift can no longer execute any writes. This usually happens
>>>>>> after
>>>>>> a few records. How we put is that we perform rowkey scans before
>>>>>> puts to
>>>>>> ensure where we last left off.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thrift eventually stops writing due to regions being offline and no
>>>>>> longer
>>>>>> serving. The regions eventually stop splitting and all the GC
>>>>>> eventually
>>>>>> complete. Once things settle back down, I am able to perform hbck
>>>>>> and fsck
>>>>>> and it again comes back with no inconsistencies
>>>>>>
>>>>>>
>>>>>>
>>>>>> I need ideas as to what is causing this
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Abraham Tom
>>>>>> Email: work2much@gmail.com
>>>>>> Phone: 415-515-3621
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Abraham Tom
>>>> Email: work2much@gmail.com
>>>> Phone: 415-515-3621
>>>>
>>>
>>>
>>
>>
>> --
>> Abraham Tom
>> Email: work2much@gmail.com
>> Phone: 415-515-3621
>>
>
>
--
Abraham Tom
Email: work2much@gmail.com
Phone: 415-515-3621
Re: hbase regions behaving badly
Posted by Ted Yu <yu...@gmail.com>.
bq. 2015-11-11 12:05:35,179 DEBUG
org.apache.hadoop.hdfs.BlockReaderFactory: Closed potentially stale remote
peer
Have you checked hdfs health ?
Cheers
On Wed, Nov 11, 2015 at 10:10 AM, Abraham Tom <wo...@gmail.com> wrote:
> Thanks
>
> I also restarted in debug mode
> and found that hbase is renewing my lease to other nodes rather quickly
> logs here
> http://pastebin.com/4EgsXCDd
>
>
> On Wed, Nov 11, 2015 at 6:20 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> Please note that IncreasingToUpperBoundRegionSplitPolicy was in effect.
>>
>> See http://hbase.apache.org/book.html#_custom_split_policies for related
>> information (there is link to
>> http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.html
>> ).
>>
>> In the future, please consider using pastebin etc. for server log so that
>> the message is concise.
>>
>> Cheers
>>
>> On Tue, Nov 10, 2015 at 10:17 PM, Abraham Tom <wo...@gmail.com>
>> wrote:
>>
>>> my hbase-site.xml snippet
>>>
>>>
>>> <property><name>hbase.mob.sweep.tool.compaction.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
>>>
>>> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
>>>
>>> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value><source>hbase-default.xml</source></property>
>>>
>>> <property><name>hbase.mob.sweep.tool.compaction.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
>>>
>>> <property><name>dfs.blockreport.split.threshold</name><value>1000000</value><source>hdfs-default.xml</source></property>
>>>
>>> <property><name>hbase.regionserver.regionSplitLimit</name><value>1000</value><source>hbase-default.xml</source></property>
>>>
>>> <property><name>hbase.splitlog.manager.timeout</name><value>120000</value><source>hbase-site.xml</source></property>
>>>
>>> <property><name>hbase.regionserver.region.split.policy</name><value>org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy</value><source>hbase-default.xml</source></property>
>>>
>>> <property><name>mapreduce.job.split.metainfo.maxsize</name><value>10000000</value><source>mapred-default.xml</source></property>
>>>
>>>
>>>
>>>
>>>
>>> and some master logs
>>> tail -100
>>> hbase-cmf-hbase-MASTER-hadoop4-private.wdc01.infra.ripple.com.log.out
>>> 2015-11-10 22:31:16,205 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {5097e5a15891c375eca53b992584c52e state=PENDING_OPEN, ts=1447216273284,
>>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>> {5097e5a15891c375eca53b992584c52e state=OPENING, ts=1447216276205, server=
>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>> 2015-11-10 22:31:16,206 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> d3b39413ee1bc88a100cd96564c4c420 from
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>>> 2015-11-10 22:31:16,206 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {cff8a38c23c3987fbbebec615f45f4ac state=OPENING, ts=1447216276185, server=
>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>> {cff8a38c23c3987fbbebec615f45f4ac state=OPEN, ts=1447216276206, server=
>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>> 2015-11-10 22:31:16,206 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> bc842e97c8ff7b0f3498b38510a9c1c1 from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,206 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {c73147aef08804451c26c1f8099008f2 state=PENDING_OPEN, ts=1447216273284,
>>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>> {c73147aef08804451c26c1f8099008f2 state=OPENING, ts=1447216276206, server=
>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>> 2015-11-10 22:31:16,207 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> cff8a38c23c3987fbbebec615f45f4ac from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,208 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {8a671fb436fb0d186b37211d847aa897 state=OPENING, ts=1447216276188, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {8a671fb436fb0d186b37211d847aa897 state=OPEN, ts=1447216276208, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,209 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> 8a671fb436fb0d186b37211d847aa897 from
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>>> 2015-11-10 22:31:16,209 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {e920fab780cdd6ba183bc50a8b5d8c72 state=PENDING_OPEN, ts=1447216273276,
>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING, ts=1447216276209, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,210 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {1258fdeaf4f00989a8282e5f9c215d7e state=OPENING, ts=1447216276193, server=
>>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189} to
>>> {1258fdeaf4f00989a8282e5f9c215d7e state=OPEN, ts=1447216276210, server=
>>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
>>> 2015-11-10 22:31:16,211 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> 1258fdeaf4f00989a8282e5f9c215d7e from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,213 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {02666182c37df5043acdcaca31ceb095 state=OPENING, ts=1447216276184, server=
>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>> {02666182c37df5043acdcaca31ceb095 state=OPEN, ts=1447216276213, server=
>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>> 2015-11-10 22:31:16,214 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=PENDING_OPEN, ts=1447216273284,
>>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING, ts=1447216276214, server=
>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>> 2015-11-10 22:31:16,215 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> 02666182c37df5043acdcaca31ceb095 from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,216 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {4c694bcbe9679747e738a630aec33c3c state=OPENING, ts=1447216276200, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {4c694bcbe9679747e738a630aec33c3c state=OPEN, ts=1447216276216, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,217 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> 4c694bcbe9679747e738a630aec33c3c from
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
>>> 2015-11-10 22:31:16,217 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {dc3169f842a4d19014f040c887b9acff state=PENDING_OPEN, ts=1447216273280,
>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {dc3169f842a4d19014f040c887b9acff state=OPENING, ts=1447216276217, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,219 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {8320a058823353a19f69a10da6827953 state=OPENING, ts=1447216276198, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {8320a058823353a19f69a10da6827953 state=OPEN, ts=1447216276219, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,219 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> 8320a058823353a19f69a10da6827953 from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,220 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {edfd9220fe6c2a37fe7521101d490941 state=PENDING_OPEN, ts=1447216273142,
>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {edfd9220fe6c2a37fe7521101d490941 state=OPENING, ts=1447216276220, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,220 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {1937fd49e94d42fe072a29ff7af885bb state=OPENING, ts=1447216276205, server=
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>> {1937fd49e94d42fe072a29ff7af885bb state=OPEN, ts=1447216276220, server=
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>> 2015-11-10 22:31:16,220 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {e4b861ca490c05fa4276e73e629b471a state=OPENING, ts=1447216276204, server=
>>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189} to
>>> {e4b861ca490c05fa4276e73e629b471a state=OPEN, ts=1447216276220, server=
>>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
>>> 2015-11-10 22:31:16,221 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> 1937fd49e94d42fe072a29ff7af885bb from
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
>>> 2015-11-10 22:31:16,221 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {9ecf375c160ffe4616b8c4fb45ea8fba state=PENDING_OPEN, ts=1447216273274,
>>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>> {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING, ts=1447216276221, server=
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>> 2015-11-10 22:31:16,221 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> e4b861ca490c05fa4276e73e629b471a from
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>>> 2015-11-10 22:31:16,222 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {2c7572eeff0a72eceec10e0301d344a7 state=OPENING, ts=1447216276205, server=
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>> {2c7572eeff0a72eceec10e0301d344a7 state=OPEN, ts=1447216276222, server=
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>> 2015-11-10 22:31:16,222 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> 2c7572eeff0a72eceec10e0301d344a7 from
>>> hadoop3-private.wdc01.infra.ripple.com,60020,1447208454196
>>> 2015-11-10 22:31:16,223 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {cb7e639dbb58b1e47c1da538165c503c state=PENDING_OPEN, ts=1447216273288,
>>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>> {cb7e639dbb58b1e47c1da538165c503c state=OPENING, ts=1447216276223, server=
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>> 2015-11-10 22:31:16,225 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {5097e5a15891c375eca53b992584c52e state=OPENING, ts=1447216276205, server=
>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>> {5097e5a15891c375eca53b992584c52e state=OPEN, ts=1447216276225, server=
>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>> 2015-11-10 22:31:16,225 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {c73147aef08804451c26c1f8099008f2 state=OPENING, ts=1447216276206, server=
>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>> {c73147aef08804451c26c1f8099008f2 state=OPEN, ts=1447216276225, server=
>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>> 2015-11-10 22:31:16,226 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {abddf225d895bd30e11205390fedc03e state=OPENING, ts=1447216276205, server=
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>> {abddf225d895bd30e11205390fedc03e state=OPEN, ts=1447216276226, server=
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>> 2015-11-10 22:31:16,226 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> 5097e5a15891c375eca53b992584c52e from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,226 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> c73147aef08804451c26c1f8099008f2 from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,227 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> abddf225d895bd30e11205390fedc03e from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,227 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {a5e6796ae343481b92b0180d817c1e17 state=PENDING_OPEN, ts=1447216273288,
>>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>> {a5e6796ae343481b92b0180d817c1e17 state=OPENING, ts=1447216276227, server=
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>> 2015-11-10 22:31:16,229 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING, ts=1447216276214, server=
>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to
>>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPEN, ts=1447216276229, server=
>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>>> 2015-11-10 22:31:16,230 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING, ts=1447216276209, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {e920fab780cdd6ba183bc50a8b5d8c72 state=OPEN, ts=1447216276230, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,230 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> dae9fb9b19a8d2076fd4be2b95aefe45 from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,230 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> e920fab780cdd6ba183bc50a8b5d8c72 from
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>>> 2015-11-10 22:31:16,230 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {2c04351380cc266df609ab53ef0e56ad state=PENDING_OPEN, ts=1447216273276,
>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {2c04351380cc266df609ab53ef0e56ad state=OPENING, ts=1447216276230, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,236 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {edfd9220fe6c2a37fe7521101d490941 state=OPENING, ts=1447216276220, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {edfd9220fe6c2a37fe7521101d490941 state=OPEN, ts=1447216276236, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,236 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING, ts=1447216276221, server=
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>> {9ecf375c160ffe4616b8c4fb45ea8fba state=OPEN, ts=1447216276236, server=
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>> 2015-11-10 22:31:16,237 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {dc3169f842a4d19014f040c887b9acff state=OPENING, ts=1447216276217, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {dc3169f842a4d19014f040c887b9acff state=OPEN, ts=1447216276237, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,240 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> edfd9220fe6c2a37fe7521101d490941 from
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
>>> 2015-11-10 22:31:16,240 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=PENDING_OPEN, ts=1447216273280,
>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING, ts=1447216276240, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,241 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {c2a9f77791512281e4491e0797e824e8 state=PENDING_OPEN, ts=1447216273277,
>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {c2a9f77791512281e4491e0797e824e8 state=OPENING, ts=1447216276241, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,241 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> dc3169f842a4d19014f040c887b9acff from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,241 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> 9ecf375c160ffe4616b8c4fb45ea8fba from
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>>> 2015-11-10 22:31:16,244 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {cb7e639dbb58b1e47c1da538165c503c state=OPENING, ts=1447216276223, server=
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>> {cb7e639dbb58b1e47c1da538165c503c state=OPEN, ts=1447216276244, server=
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>> 2015-11-10 22:31:16,245 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> cb7e639dbb58b1e47c1da538165c503c from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,246 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {a5e6796ae343481b92b0180d817c1e17 state=OPENING, ts=1447216276227, server=
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to
>>> {a5e6796ae343481b92b0180d817c1e17 state=OPEN, ts=1447216276246, server=
>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>>> 2015-11-10 22:31:16,246 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> a5e6796ae343481b92b0180d817c1e17 from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,252 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {2c04351380cc266df609ab53ef0e56ad state=OPENING, ts=1447216276230, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {2c04351380cc266df609ab53ef0e56ad state=OPEN, ts=1447216276252, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,253 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> 2c04351380cc266df609ab53ef0e56ad from
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>>> 2015-11-10 22:31:16,253 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {15701b8131ca505f6cce148d49cfbeee state=PENDING_OPEN, ts=1447216273280,
>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {15701b8131ca505f6cce148d49cfbeee state=OPENING, ts=1447216276253, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,257 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING, ts=1447216276240, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPEN, ts=1447216276257, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,257 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {c2a9f77791512281e4491e0797e824e8 state=OPENING, ts=1447216276241, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {c2a9f77791512281e4491e0797e824e8 state=OPEN, ts=1447216276257, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,258 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> a4f70c6c01f31f6d3d65e0cf55200c80 from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,258 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> c2a9f77791512281e4491e0797e824e8 from
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>>> 2015-11-10 22:31:16,258 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {7ce823a4e0f56bf8c854175409bb17b6 state=PENDING_OPEN, ts=1447216273280,
>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING, ts=1447216276258, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,258 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {1fbbd37bdff125be32d352c98dffd2b1 state=PENDING_OPEN, ts=1447216273280,
>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING, ts=1447216276258, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,280 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING, ts=1447216276258, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {1fbbd37bdff125be32d352c98dffd2b1 state=OPEN, ts=1447216276280, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,280 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING, ts=1447216276258, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {7ce823a4e0f56bf8c854175409bb17b6 state=OPEN, ts=1447216276280, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,281 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> 1fbbd37bdff125be32d352c98dffd2b1 from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,281 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> 7ce823a4e0f56bf8c854175409bb17b6 from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,281 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {cc52e0ba23708df10d696d0e33d9d137 state=PENDING_OPEN, ts=1447216273280,
>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {cc52e0ba23708df10d696d0e33d9d137 state=OPENING, ts=1447216276281, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,281 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {21b24baeaf26e12d02f79f243076850a state=PENDING_OPEN, ts=1447216273280,
>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {21b24baeaf26e12d02f79f243076850a state=OPENING, ts=1447216276281, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,306 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {15701b8131ca505f6cce148d49cfbeee state=OPENING, ts=1447216276253, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {15701b8131ca505f6cce148d49cfbeee state=OPEN, ts=1447216276306, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,307 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> 15701b8131ca505f6cce148d49cfbeee from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,325 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {cc52e0ba23708df10d696d0e33d9d137 state=OPENING, ts=1447216276281, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {cc52e0ba23708df10d696d0e33d9d137 state=OPEN, ts=1447216276325, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,325 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Transition
>>> {21b24baeaf26e12d02f79f243076850a state=OPENING, ts=1447216276281, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to
>>> {21b24baeaf26e12d02f79f243076850a state=OPEN, ts=1447216276325, server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>>> 2015-11-10 22:31:16,326 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> cc52e0ba23708df10d696d0e33d9d137 from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:31:16,326 INFO
>>> org.apache.hadoop.hbase.master.RegionStates: Offlined
>>> 21b24baeaf26e12d02f79f243076850a from
>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>>> 2015-11-10 22:36:01,090 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 22:41:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 22:46:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=89, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 22:47:34,680 INFO
>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>>> 10.125.122.237 set balanceSwitch=false
>>> 2015-11-10 22:47:41,845 INFO
>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>>> 10.125.122.237 set balanceSwitch=true
>>> 2015-11-10 22:51:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 22:56:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=149, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 23:01:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=179, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 23:06:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=209, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 23:11:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=239, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 23:16:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=269, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 23:17:34,662 INFO
>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>>> 10.125.122.237 set balanceSwitch=false
>>> 2015-11-10 23:17:38,707 INFO
>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>>> 10.125.122.237 set balanceSwitch=true
>>> 2015-11-10 23:21:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=299, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 23:26:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=329, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 23:31:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=359, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 23:31:01,096 INFO org.apache.hadoop.hbase.mob.MobFileCache:
>>> MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted
>>> files: 0
>>> 2015-11-10 23:36:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=389, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 23:41:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=419, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 23:46:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=449, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 23:47:34,668 INFO
>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>>> 10.125.122.237 set balanceSwitch=false
>>> 2015-11-10 23:47:44,843 INFO
>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>>> 10.125.122.237 set balanceSwitch=true
>>> 2015-11-10 23:51:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=479, evicted=0, evictedPerRun=0.0
>>> 2015-11-10 23:56:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=509, evicted=0, evictedPerRun=0.0
>>> 2015-11-11 00:01:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=539, evicted=0, evictedPerRun=0.0
>>> 2015-11-11 00:06:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=569, evicted=0, evictedPerRun=0.0
>>> 2015-11-11 00:11:01,089 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>>> cachingHitsRatio=0,evictions=599, evicted=0, evictedPerRun=0.0
>>>
>>> Region server logs
>>> tail -100
>>> hbase-cmf-hbase-REGIONSERVER-hadoop4-private.wdc01.infra.ripple.com.log.out
>>> 2015-11-10 22:31:16,255 INFO
>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>> cacheDataCompressed=false, prefetchOnOpen=false
>>> 2015-11-10 22:31:16,255 INFO
>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>> major jitter 0.500000, min locality to compact 0.000000
>>> 2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>>> Updated row prod_control,,1434077692105.a4f70c6c01f31f6d3d65e0cf55200c80.
>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>>> 2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>>> Updated row
>>> prod_agg_account_payments,20140325000000|rH48e8bf5kNWuFRBwgm5sCH133cd9zgV29,1447197624013.c2a9f77791512281e4491e0797e824e8.
>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>>> 2015-11-10 22:31:16,259 INFO
>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>> cacheDataCompressed=false, prefetchOnOpen=false
>>> 2015-11-10 22:31:16,259 INFO
>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>> cacheDataCompressed=false, prefetchOnOpen=false
>>> 2015-11-10 22:31:16,259 INFO
>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>> major jitter 0.500000, min locality to compact 0.000000
>>> 2015-11-10 22:31:16,259 INFO
>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>> major jitter 0.500000, min locality to compact 0.000000
>>> 2015-11-10 22:31:16,262 INFO
>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>> cacheDataCompressed=false, prefetchOnOpen=false
>>> 2015-11-10 22:31:16,262 INFO
>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>> major jitter 0.500000, min locality to compact 0.000000
>>> 2015-11-10 22:31:16,265 INFO
>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>> cacheDataCompressed=false, prefetchOnOpen=false
>>> 2015-11-10 22:31:16,265 INFO
>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>> major jitter 0.500000, min locality to compact 0.000000
>>> 2015-11-10 22:31:16,265 INFO
>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>> cacheDataCompressed=false, prefetchOnOpen=false
>>> 2015-11-10 22:31:16,265 INFO
>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>> major jitter 0.500000, min locality to compact 0.000000
>>> 2015-11-10 22:31:16,267 INFO
>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>> cacheDataCompressed=false, prefetchOnOpen=false
>>> 2015-11-10 22:31:16,268 INFO
>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>> major jitter 0.500000, min locality to compact 0.000000
>>> 2015-11-10 22:31:16,272 INFO
>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>> cacheDataCompressed=false, prefetchOnOpen=false
>>> 2015-11-10 22:31:16,273 INFO
>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>> major jitter 0.500000, min locality to compact 0.000000
>>> 2015-11-10 22:31:16,274 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>>> 1fbbd37bdff125be32d352c98dffd2b1; next sequenceid=180083
>>> 2015-11-10 22:31:16,274 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>>> 7ce823a4e0f56bf8c854175409bb17b6; next sequenceid=21535
>>> 2015-11-10 22:31:16,275 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>>> for
>>> prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1.
>>> 2015-11-10 22:31:16,276 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>>> for
>>> prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6.
>>> 2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>>> Updated row
>>> prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1.
>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>>> 2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>>> Updated row
>>> prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6.
>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>>> 2015-11-10 22:31:16,282 INFO
>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>> cacheDataCompressed=false, prefetchOnOpen=false
>>> 2015-11-10 22:31:16,282 INFO
>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>> cacheDataCompressed=false, prefetchOnOpen=false
>>> 2015-11-10 22:31:16,282 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>>> 15701b8131ca505f6cce148d49cfbeee; next sequenceid=1029
>>> 2015-11-10 22:31:16,283 INFO
>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>> major jitter 0.500000, min locality to compact 0.000000
>>> 2015-11-10 22:31:16,283 INFO
>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>> major jitter 0.500000, min locality to compact 0.000000
>>> 2015-11-10 22:31:16,284 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>>> for
>>> prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee.
>>> 2015-11-10 22:31:16,304 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>>> Updated row
>>> prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee.
>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>>> 2015-11-10 22:31:16,308 INFO
>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>> cacheDataCompressed=false, prefetchOnOpen=false
>>> 2015-11-10 22:31:16,308 INFO
>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>> major jitter 0.500000, min locality to compact 0.000000
>>> 2015-11-10 22:31:16,308 INFO
>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>> cacheDataCompressed=false, prefetchOnOpen=false
>>> 2015-11-10 22:31:16,309 INFO
>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>>> major jitter 0.500000, min locality to compact 0.000000
>>> 2015-11-10 22:31:16,318 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>>> 21b24baeaf26e12d02f79f243076850a; next sequenceid=114628
>>> 2015-11-10 22:31:16,318 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>>> cc52e0ba23708df10d696d0e33d9d137; next sequenceid=161394
>>> 2015-11-10 22:31:16,320 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>>> for
>>> prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a.
>>> 2015-11-10 22:31:16,320 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>>> for prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD|2014111212
>>> ,1447198690089.cc52e0ba23708df10d696d0e33d9d137.
>>> 2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>>> Updated row prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD|
>>> 2014111212,1447198690089.cc52e0ba23708df10d696d0e33d9d137. with server=
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>>> 2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>>> Updated row
>>> prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a.
>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>>> 2015-11-10 22:31:21,184 WARN org.apache.hadoop.metrics2.lib.Interns:
>>> Metrics intern cache overflow at 2011 for
>>> RegionServer={RegionServer=MetricsInfoImpl{name=RegionServer,
>>> description=RegionServer}, RegionServer
>>> record=MetricsInfoImpl{name=RegionServer, description=RegionServer record}}
>>> 2015-11-10 22:31:22,377 WARN
>>> org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil:
>>> hbase.regionserver.global.memstore.upperLimit is deprecated by
>>> hbase.regionserver.global.memstore.size
>>> 2015-11-10 22:31:22,403 INFO
>>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3
>>> (all) file(s) in f of
>>> prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041.
>>> into d98861e6b5364d819d11c3f3e1c8fb56(size=180.4 M), total size for store
>>> is 180.4 M. This selection was in queue for 0sec, and took 7sec to execute.
>>> 2015-11-10 22:31:22,405 INFO
>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>>> compaction: Request =
>>> regionName=prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041.,
>>> storeName=f, fileCount=3, fileSize=180.4 M, priority=27,
>>> time=741524299592168; duration=7sec
>>> 2015-11-10 22:31:22,413 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on f in
>>> region
>>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
>>> 2015-11-10 22:31:22,413 INFO
>>> org.apache.hadoop.hbase.regionserver.HStore: Starting compaction of 3
>>> file(s) in f of
>>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
>>> into tmpdir=hdfs://
>>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_transactions/72f5a6c7dba1e486eae6b75f4e9f0d5d/.tmp,
>>> totalSize=82.2 M
>>> 2015-11-10 22:31:22,416 INFO
>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>> cacheDataCompressed=false, prefetchOnOpen=false
>>> 2015-11-10 22:31:25,301 INFO
>>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3
>>> (all) file(s) in f of
>>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
>>> into f7678de8dd8149428938a0aedfcbcaf9(size=82.2 M), total size for store is
>>> 82.2 M. This selection was in queue for 0sec, and took 2sec to execute.
>>> 2015-11-10 22:31:25,302 INFO
>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>>> compaction: Request =
>>> regionName=prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.,
>>> storeName=f, fileCount=3, fileSize=82.2 M, priority=27,
>>> time=741532147583369; duration=2sec
>>> 2015-11-10 22:31:25,305 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on f in
>>> region
>>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
>>> 2015-11-10 22:31:25,305 INFO
>>> org.apache.hadoop.hbase.regionserver.HStore: Starting compaction of 3
>>> file(s) in f of
>>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
>>> into tmpdir=hdfs://
>>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_offers/05284b692219df79852f3deaa2ce1927/.tmp,
>>> totalSize=218.4 M
>>> 2015-11-10 22:31:25,310 INFO
>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>> cacheDataCompressed=false, prefetchOnOpen=false
>>> 2015-11-10 22:31:39,481 INFO
>>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3
>>> (all) file(s) in f of
>>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
>>> into 4aea6e4301f848ff9bf727f4e511978a(size=214.6 M), total size for store
>>> is 214.6 M. This selection was in queue for 0sec, and took 14sec to execute.
>>> 2015-11-10 22:31:39,481 INFO
>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>>> compaction: Request =
>>> regionName=prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.,
>>> storeName=f, fileCount=3, fileSize=218.4 M, priority=27,
>>> time=741535039619615; duration=14sec
>>> 2015-11-10 22:31:39,484 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on d in
>>> region
>>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
>>> 2015-11-10 22:31:39,484 INFO
>>> org.apache.hadoop.hbase.regionserver.HStore: Starting compaction of 3
>>> file(s) in d of
>>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
>>> into tmpdir=hdfs://
>>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_balance_changes/40d39088cc3efe2d230a7d6f8d7b205b/.tmp,
>>> totalSize=197.7 M
>>> 2015-11-10 22:31:39,490 INFO
>>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>>> cacheDataCompressed=false, prefetchOnOpen=false
>>> 2015-11-10 22:31:52,393 INFO
>>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3
>>> (all) file(s) in d of
>>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
>>> into 1e48a57cdb544e898e78c9e8276ec05b(size=197.7 M), total size for store
>>> is 197.7 M. This selection was in queue for 0sec, and took 12sec to execute.
>>> 2015-11-10 22:31:52,393 INFO
>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>>> compaction: Request =
>>> regionName=prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.,
>>> storeName=d, fileCount=3, fileSize=197.7 M, priority=27,
>>> time=741549219351203; duration=12sec
>>> 2015-11-10 22:32:01,402 INFO
>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Atomically
>>> moving hadoop3-private.wdc01.infra.ripple.com,60020,1447102836904's
>>> wals to my queue
>>> 2015-11-10 22:32:01,405 WARN
>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer 1 didn't
>>> exist, skipping the replay
>>> 2015-11-10 22:32:01,406 WARN
>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer
>>> 1-hadoop4-private.wdc01.infra.ripple.com,60020,1447102248964 didn't
>>> exist, skipping the replay
>>> 2015-11-10 22:32:01,412 WARN
>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Got exception
>>> in copyQueuesFromRSUsingMulti:
>>> org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode
>>> = Directory not empty
>>> at
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
>>> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1152)
>>> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:1118)
>>> at
>>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:685)
>>> at
>>> org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1550)
>>> at
>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.copyQueuesFromRSUsingMulti(ReplicationQueuesZKImpl.java:340)
>>> at
>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.claimQueues(ReplicationQueuesZKImpl.java:172)
>>> at
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:570)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>> 2015-11-10 22:36:01,223 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=3.78 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=4, accesses=63330, hits=6,
>>> hitRatio=0.01%, , cachingAccesses=21, cachingHits=4,
>>> cachingHitsRatio=19.05%, evictions=29, evicted=3,
>>> evictedPerRun=0.1034482792019844
>>> 2015-11-10 22:41:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.15 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=7, accesses=63856, hits=8,
>>> hitRatio=0.01%, , cachingAccesses=32, cachingHits=6,
>>> cachingHitsRatio=18.75%, evictions=59, evicted=3,
>>> evictedPerRun=0.050847455859184265
>>> 2015-11-10 22:46:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
>>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
>>> cachingHitsRatio=21.43%, evictions=89, evicted=3,
>>> evictedPerRun=0.033707864582538605
>>> 2015-11-10 22:51:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
>>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
>>> cachingHitsRatio=21.43%, evictions=119, evicted=3,
>>> evictedPerRun=0.02521008439362049
>>> 2015-11-10 22:56:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
>>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
>>> cachingHitsRatio=21.43%, evictions=149, evicted=3,
>>> evictedPerRun=0.020134227350354195
>>> 2015-11-10 23:01:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63872, hits=13,
>>> hitRatio=0.02%, , cachingAccesses=48, cachingHits=11,
>>> cachingHitsRatio=22.92%, evictions=179, evicted=3,
>>> evictedPerRun=0.016759777441620827
>>> 2015-11-10 23:06:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63890, hits=21,
>>> hitRatio=0.03%, , cachingAccesses=66, cachingHits=19,
>>> cachingHitsRatio=28.79%, evictions=209, evicted=3,
>>> evictedPerRun=0.014354066923260689
>>> 2015-11-10 23:11:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64416, hits=26,
>>> hitRatio=0.04%, , cachingAccesses=77, cachingHits=24,
>>> cachingHitsRatio=31.17%, evictions=239, evicted=3,
>>> evictedPerRun=0.012552301399409771
>>> 2015-11-10 23:16:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
>>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
>>> cachingHitsRatio=33.33%, evictions=269, evicted=3,
>>> evictedPerRun=0.011152416467666626
>>> 2015-11-10 23:21:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
>>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
>>> cachingHitsRatio=33.33%, evictions=299, evicted=3,
>>> evictedPerRun=0.010033444501459599
>>> 2015-11-10 23:26:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
>>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
>>> cachingHitsRatio=33.33%, evictions=329, evicted=3,
>>> evictedPerRun=0.009118541143834591
>>> 2015-11-10 23:31:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64432, hits=34,
>>> hitRatio=0.05%, , cachingAccesses=93, cachingHits=32,
>>> cachingHitsRatio=34.41%, evictions=359, evicted=3,
>>> evictedPerRun=0.008356546051800251
>>> 2015-11-10 23:31:01,228 INFO org.apache.hadoop.hbase.mob.MobFileCache:
>>> MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted
>>> files: 0
>>> 2015-11-10 23:31:06,790 INFO
>>> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Rolled WAL /hbase/WALs/
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/
>>> hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
>>> with entries=247, filesize=125.29 KB; new WAL /hbase/WALs/
>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/
>>> hadoop4-private.wdc01.infra.ripple.com
>>> %2C60020%2C1447216261115.null0.1447219866745
>>> 2015-11-10 23:31:06,792 INFO
>>> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Archiving hdfs://
>>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/WALs/hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
>>> to hdfs://
>>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/oldWALs/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
>>> 2015-11-10 23:36:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64450, hits=42,
>>> hitRatio=0.07%, , cachingAccesses=111, cachingHits=40,
>>> cachingHitsRatio=36.04%, evictions=389, evicted=3,
>>> evictedPerRun=0.007712082471698523
>>> 2015-11-10 23:41:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64976, hits=47,
>>> hitRatio=0.07%, , cachingAccesses=122, cachingHits=45,
>>> cachingHitsRatio=36.89%, evictions=419, evicted=3,
>>> evictedPerRun=0.0071599045768380165
>>> 2015-11-10 23:46:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
>>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
>>> cachingHitsRatio=37.88%, evictions=449, evicted=3,
>>> evictedPerRun=0.00668151443824172
>>> 2015-11-10 23:51:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
>>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
>>> cachingHitsRatio=37.88%, evictions=479, evicted=3,
>>> evictedPerRun=0.00626304792240262
>>> 2015-11-10 23:56:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
>>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
>>> cachingHitsRatio=37.88%, evictions=509, evicted=3,
>>> evictedPerRun=0.005893909838050604
>>> 2015-11-11 00:01:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64992, hits=55,
>>> hitRatio=0.08%, , cachingAccesses=138, cachingHits=53,
>>> cachingHitsRatio=38.41%, evictions=539, evicted=3,
>>> evictedPerRun=0.005565862637013197
>>> 2015-11-11 00:06:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65010, hits=63,
>>> hitRatio=0.10%, , cachingAccesses=156, cachingHits=61,
>>> cachingHitsRatio=39.10%, evictions=569, evicted=3,
>>> evictedPerRun=0.005272407550364733
>>> 2015-11-11 00:11:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65536, hits=68,
>>> hitRatio=0.10%, , cachingAccesses=167, cachingHits=66,
>>> cachingHitsRatio=39.52%, evictions=599, evicted=3,
>>> evictedPerRun=0.00500834733247757
>>> 2015-11-11 00:16:01,222 INFO
>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65546, hits=73,
>>> hitRatio=0.11%, , cachingAccesses=177, cachingHits=71,
>>> cachingHitsRatio=40.11%, evictions=629, evicted=3,
>>> evictedPerRun=0.004769475199282169
>>>
>>>
>>>
>>>
>>> On Tue, Nov 10, 2015 at 8:15 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> bq. The region exits are forcing the data to start transferring to
>>>> other regions,
>>>>
>>>> Can you elaborate a bit more on the above ? Did you mean some regions
>>>> were transferred to other region server(s) ?
>>>>
>>>> Can you determined the effective RegionSplitPolicy ?
>>>>
>>>> Snippets of server logs would help illustrate the situation.
>>>>
>>>> Thanks
>>>>
>>>> On Tue, Nov 10, 2015 at 8:09 PM, Abraham Tom <wo...@gmail.com>
>>>> wrote:
>>>>
>>>>> I am on CDH 5.4.8 hbase 1.0
>>>>>
>>>>> I am migrating to a new cluster and the data is sourced from CDH 5.3
>>>>> hbase
>>>>> 0.98.6
>>>>>
>>>>> I took snapshots from the old cluster exported them to the new cluster
>>>>> - no
>>>>> errors
>>>>>
>>>>> cloned the table, flush, major_compact and balanced the regions
>>>>>
>>>>> the configs are practically identical to the old cluster
>>>>>
>>>>>
>>>>>
>>>>> hbck and fsck all check out with no inconsistencies found
>>>>>
>>>>>
>>>>>
>>>>> yet when we try to load data via thrift the regions start splitting at
>>>>> an
>>>>> outrageous pace.
>>>>>
>>>>> Eventually the regions become offline starting a snowball affect. The
>>>>> region exits are forcing the data to start transferring to other
>>>>> regions,
>>>>> with that creates large GC activity and multiple regions in transition
>>>>> before thrift can no longer execute any writes. This usually happens
>>>>> after
>>>>> a few records. How we put is that we perform rowkey scans before
>>>>> puts to
>>>>> ensure where we last left off.
>>>>>
>>>>>
>>>>>
>>>>> Thrift eventually stops writing due to regions being offline and no
>>>>> longer
>>>>> serving. The regions eventually stop splitting and all the GC
>>>>> eventually
>>>>> complete. Once things settle back down, I am able to perform hbck and
>>>>> fsck
>>>>> and it again comes back with no inconsistencies
>>>>>
>>>>>
>>>>>
>>>>> I need ideas as to what is causing this
>>>>>
>>>>>
>>>>> --
>>>>> Abraham Tom
>>>>> Email: work2much@gmail.com
>>>>> Phone: 415-515-3621
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Abraham Tom
>>> Email: work2much@gmail.com
>>> Phone: 415-515-3621
>>>
>>
>>
>
>
> --
> Abraham Tom
> Email: work2much@gmail.com
> Phone: 415-515-3621
>
Re: hbase regions behaving badly
Posted by Abraham Tom <wo...@gmail.com>.
Thanks
I also restarted in debug mode
and found that hbase is renewing my lease to other nodes rather quickly
logs here
http://pastebin.com/4EgsXCDd
On Wed, Nov 11, 2015 at 6:20 AM, Ted Yu <yu...@gmail.com> wrote:
> Please note that IncreasingToUpperBoundRegionSplitPolicy was in effect.
>
> See http://hbase.apache.org/book.html#_custom_split_policies for related
> information (there is link to
> http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.html
> ).
>
> In the future, please consider using pastebin etc. for server log so that
> the message is concise.
>
> Cheers
>
> On Tue, Nov 10, 2015 at 10:17 PM, Abraham Tom <wo...@gmail.com> wrote:
>
>> my hbase-site.xml snippet
>>
>>
>> <property><name>hbase.mob.sweep.tool.compaction.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
>>
>> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
>>
>> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value><source>hbase-default.xml</source></property>
>>
>> <property><name>hbase.mob.sweep.tool.compaction.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
>>
>> <property><name>dfs.blockreport.split.threshold</name><value>1000000</value><source>hdfs-default.xml</source></property>
>>
>> <property><name>hbase.regionserver.regionSplitLimit</name><value>1000</value><source>hbase-default.xml</source></property>
>>
>> <property><name>hbase.splitlog.manager.timeout</name><value>120000</value><source>hbase-site.xml</source></property>
>>
>> <property><name>hbase.regionserver.region.split.policy</name><value>org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy</value><source>hbase-default.xml</source></property>
>>
>> <property><name>mapreduce.job.split.metainfo.maxsize</name><value>10000000</value><source>mapred-default.xml</source></property>
>>
>>
>>
>>
>>
>> and some master logs
>> tail -100
>> hbase-cmf-hbase-MASTER-hadoop4-private.wdc01.infra.ripple.com.log.out
>> 2015-11-10 22:31:16,205 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {5097e5a15891c375eca53b992584c52e state=PENDING_OPEN,
>> ts=1447216273284, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {5097e5a15891c375eca53b992584c52e state=OPENING, ts=1447216276205,
>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined d3b39413ee1bc88a100cd96564c4c420 from
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>> 2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {cff8a38c23c3987fbbebec615f45f4ac state=OPENING,
>> ts=1447216276185, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {cff8a38c23c3987fbbebec615f45f4ac state=OPEN, ts=1447216276206, server=
>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined bc842e97c8ff7b0f3498b38510a9c1c1 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {c73147aef08804451c26c1f8099008f2 state=PENDING_OPEN,
>> ts=1447216273284, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {c73147aef08804451c26c1f8099008f2 state=OPENING, ts=1447216276206,
>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,207 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined cff8a38c23c3987fbbebec615f45f4ac from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,208 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {8a671fb436fb0d186b37211d847aa897 state=OPENING,
>> ts=1447216276188, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {8a671fb436fb0d186b37211d847aa897 state=OPEN, ts=1447216276208, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,209 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 8a671fb436fb0d186b37211d847aa897 from
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>> 2015-11-10 22:31:16,209 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {e920fab780cdd6ba183bc50a8b5d8c72 state=PENDING_OPEN,
>> ts=1447216273276, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING, ts=1447216276209,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,210 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {1258fdeaf4f00989a8282e5f9c215d7e state=OPENING,
>> ts=1447216276193, server=hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
>> to {1258fdeaf4f00989a8282e5f9c215d7e state=OPEN, ts=1447216276210, server=
>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
>> 2015-11-10 22:31:16,211 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 1258fdeaf4f00989a8282e5f9c215d7e from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,213 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {02666182c37df5043acdcaca31ceb095 state=OPENING,
>> ts=1447216276184, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {02666182c37df5043acdcaca31ceb095 state=OPEN, ts=1447216276213, server=
>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,214 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {dae9fb9b19a8d2076fd4be2b95aefe45 state=PENDING_OPEN,
>> ts=1447216273284, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING, ts=1447216276214,
>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,215 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 02666182c37df5043acdcaca31ceb095 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,216 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {4c694bcbe9679747e738a630aec33c3c state=OPENING,
>> ts=1447216276200, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {4c694bcbe9679747e738a630aec33c3c state=OPEN, ts=1447216276216, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,217 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 4c694bcbe9679747e738a630aec33c3c from
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
>> 2015-11-10 22:31:16,217 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {dc3169f842a4d19014f040c887b9acff state=PENDING_OPEN,
>> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {dc3169f842a4d19014f040c887b9acff state=OPENING, ts=1447216276217,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,219 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {8320a058823353a19f69a10da6827953 state=OPENING,
>> ts=1447216276198, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {8320a058823353a19f69a10da6827953 state=OPEN, ts=1447216276219, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,219 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 8320a058823353a19f69a10da6827953 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,220 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {edfd9220fe6c2a37fe7521101d490941 state=PENDING_OPEN,
>> ts=1447216273142, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {edfd9220fe6c2a37fe7521101d490941 state=OPENING, ts=1447216276220,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,220 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {1937fd49e94d42fe072a29ff7af885bb state=OPENING,
>> ts=1447216276205, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {1937fd49e94d42fe072a29ff7af885bb state=OPEN, ts=1447216276220, server=
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,220 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {e4b861ca490c05fa4276e73e629b471a state=OPENING,
>> ts=1447216276204, server=hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
>> to {e4b861ca490c05fa4276e73e629b471a state=OPEN, ts=1447216276220, server=
>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
>> 2015-11-10 22:31:16,221 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 1937fd49e94d42fe072a29ff7af885bb from
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
>> 2015-11-10 22:31:16,221 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {9ecf375c160ffe4616b8c4fb45ea8fba state=PENDING_OPEN,
>> ts=1447216273274, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING, ts=1447216276221,
>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,221 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined e4b861ca490c05fa4276e73e629b471a from
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>> 2015-11-10 22:31:16,222 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {2c7572eeff0a72eceec10e0301d344a7 state=OPENING,
>> ts=1447216276205, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {2c7572eeff0a72eceec10e0301d344a7 state=OPEN, ts=1447216276222, server=
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,222 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 2c7572eeff0a72eceec10e0301d344a7 from
>> hadoop3-private.wdc01.infra.ripple.com,60020,1447208454196
>> 2015-11-10 22:31:16,223 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {cb7e639dbb58b1e47c1da538165c503c state=PENDING_OPEN,
>> ts=1447216273288, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {cb7e639dbb58b1e47c1da538165c503c state=OPENING, ts=1447216276223,
>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,225 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {5097e5a15891c375eca53b992584c52e state=OPENING,
>> ts=1447216276205, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {5097e5a15891c375eca53b992584c52e state=OPEN, ts=1447216276225, server=
>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,225 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {c73147aef08804451c26c1f8099008f2 state=OPENING,
>> ts=1447216276206, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {c73147aef08804451c26c1f8099008f2 state=OPEN, ts=1447216276225, server=
>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,226 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {abddf225d895bd30e11205390fedc03e state=OPENING,
>> ts=1447216276205, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {abddf225d895bd30e11205390fedc03e state=OPEN, ts=1447216276226, server=
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,226 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 5097e5a15891c375eca53b992584c52e from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,226 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined c73147aef08804451c26c1f8099008f2 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,227 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined abddf225d895bd30e11205390fedc03e from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,227 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {a5e6796ae343481b92b0180d817c1e17 state=PENDING_OPEN,
>> ts=1447216273288, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {a5e6796ae343481b92b0180d817c1e17 state=OPENING, ts=1447216276227,
>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,229 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING,
>> ts=1447216276214, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPEN, ts=1447216276229, server=
>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING,
>> ts=1447216276209, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {e920fab780cdd6ba183bc50a8b5d8c72 state=OPEN, ts=1447216276230, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined dae9fb9b19a8d2076fd4be2b95aefe45 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined e920fab780cdd6ba183bc50a8b5d8c72 from
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>> 2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {2c04351380cc266df609ab53ef0e56ad state=PENDING_OPEN,
>> ts=1447216273276, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {2c04351380cc266df609ab53ef0e56ad state=OPENING, ts=1447216276230,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,236 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {edfd9220fe6c2a37fe7521101d490941 state=OPENING,
>> ts=1447216276220, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {edfd9220fe6c2a37fe7521101d490941 state=OPEN, ts=1447216276236, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,236 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING,
>> ts=1447216276221, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {9ecf375c160ffe4616b8c4fb45ea8fba state=OPEN, ts=1447216276236, server=
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,237 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {dc3169f842a4d19014f040c887b9acff state=OPENING,
>> ts=1447216276217, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {dc3169f842a4d19014f040c887b9acff state=OPEN, ts=1447216276237, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,240 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined edfd9220fe6c2a37fe7521101d490941 from
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
>> 2015-11-10 22:31:16,240 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {a4f70c6c01f31f6d3d65e0cf55200c80 state=PENDING_OPEN,
>> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING, ts=1447216276240,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,241 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {c2a9f77791512281e4491e0797e824e8 state=PENDING_OPEN,
>> ts=1447216273277, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {c2a9f77791512281e4491e0797e824e8 state=OPENING, ts=1447216276241,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,241 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined dc3169f842a4d19014f040c887b9acff from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,241 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 9ecf375c160ffe4616b8c4fb45ea8fba from
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>> 2015-11-10 22:31:16,244 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {cb7e639dbb58b1e47c1da538165c503c state=OPENING,
>> ts=1447216276223, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {cb7e639dbb58b1e47c1da538165c503c state=OPEN, ts=1447216276244, server=
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,245 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined cb7e639dbb58b1e47c1da538165c503c from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,246 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {a5e6796ae343481b92b0180d817c1e17 state=OPENING,
>> ts=1447216276227, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {a5e6796ae343481b92b0180d817c1e17 state=OPEN, ts=1447216276246, server=
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,246 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined a5e6796ae343481b92b0180d817c1e17 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,252 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {2c04351380cc266df609ab53ef0e56ad state=OPENING,
>> ts=1447216276230, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {2c04351380cc266df609ab53ef0e56ad state=OPEN, ts=1447216276252, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,253 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 2c04351380cc266df609ab53ef0e56ad from
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>> 2015-11-10 22:31:16,253 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {15701b8131ca505f6cce148d49cfbeee state=PENDING_OPEN,
>> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {15701b8131ca505f6cce148d49cfbeee state=OPENING, ts=1447216276253,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,257 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING,
>> ts=1447216276240, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPEN, ts=1447216276257, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,257 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {c2a9f77791512281e4491e0797e824e8 state=OPENING,
>> ts=1447216276241, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {c2a9f77791512281e4491e0797e824e8 state=OPEN, ts=1447216276257, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined a4f70c6c01f31f6d3d65e0cf55200c80 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined c2a9f77791512281e4491e0797e824e8 from
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>> 2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {7ce823a4e0f56bf8c854175409bb17b6 state=PENDING_OPEN,
>> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING, ts=1447216276258,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {1fbbd37bdff125be32d352c98dffd2b1 state=PENDING_OPEN,
>> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING, ts=1447216276258,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,280 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING,
>> ts=1447216276258, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {1fbbd37bdff125be32d352c98dffd2b1 state=OPEN, ts=1447216276280, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,280 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING,
>> ts=1447216276258, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {7ce823a4e0f56bf8c854175409bb17b6 state=OPEN, ts=1447216276280, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 1fbbd37bdff125be32d352c98dffd2b1 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 7ce823a4e0f56bf8c854175409bb17b6 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {cc52e0ba23708df10d696d0e33d9d137 state=PENDING_OPEN,
>> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {cc52e0ba23708df10d696d0e33d9d137 state=OPENING, ts=1447216276281,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {21b24baeaf26e12d02f79f243076850a state=PENDING_OPEN,
>> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {21b24baeaf26e12d02f79f243076850a state=OPENING, ts=1447216276281,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,306 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {15701b8131ca505f6cce148d49cfbeee state=OPENING,
>> ts=1447216276253, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {15701b8131ca505f6cce148d49cfbeee state=OPEN, ts=1447216276306, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,307 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 15701b8131ca505f6cce148d49cfbeee from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,325 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {cc52e0ba23708df10d696d0e33d9d137 state=OPENING,
>> ts=1447216276281, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {cc52e0ba23708df10d696d0e33d9d137 state=OPEN, ts=1447216276325, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,325 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {21b24baeaf26e12d02f79f243076850a state=OPENING,
>> ts=1447216276281, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {21b24baeaf26e12d02f79f243076850a state=OPEN, ts=1447216276325, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,326 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined cc52e0ba23708df10d696d0e33d9d137 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,326 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 21b24baeaf26e12d02f79f243076850a from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:36:01,090 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0
>> 2015-11-10 22:41:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0
>> 2015-11-10 22:46:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=89, evicted=0, evictedPerRun=0.0
>> 2015-11-10 22:47:34,680 INFO
>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>> 10.125.122.237 set balanceSwitch=false
>> 2015-11-10 22:47:41,845 INFO
>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>> 10.125.122.237 set balanceSwitch=true
>> 2015-11-10 22:51:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
>> 2015-11-10 22:56:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=149, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:01:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=179, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:06:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=209, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:11:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=239, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:16:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=269, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:17:34,662 INFO
>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>> 10.125.122.237 set balanceSwitch=false
>> 2015-11-10 23:17:38,707 INFO
>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>> 10.125.122.237 set balanceSwitch=true
>> 2015-11-10 23:21:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=299, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:26:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=329, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:31:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=359, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:31:01,096 INFO org.apache.hadoop.hbase.mob.MobFileCache:
>> MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted
>> files: 0
>> 2015-11-10 23:36:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=389, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:41:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=419, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:46:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=449, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:47:34,668 INFO
>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>> 10.125.122.237 set balanceSwitch=false
>> 2015-11-10 23:47:44,843 INFO
>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>> 10.125.122.237 set balanceSwitch=true
>> 2015-11-10 23:51:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=479, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:56:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=509, evicted=0, evictedPerRun=0.0
>> 2015-11-11 00:01:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=539, evicted=0, evictedPerRun=0.0
>> 2015-11-11 00:06:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=569, evicted=0, evictedPerRun=0.0
>> 2015-11-11 00:11:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=599, evicted=0, evictedPerRun=0.0
>>
>> Region server logs
>> tail -100
>> hbase-cmf-hbase-REGIONSERVER-hadoop4-private.wdc01.infra.ripple.com.log.out
>> 2015-11-10 22:31:16,255 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,255 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>> Updated row prod_control,,1434077692105.a4f70c6c01f31f6d3d65e0cf55200c80.
>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>> 2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>> Updated row
>> prod_agg_account_payments,20140325000000|rH48e8bf5kNWuFRBwgm5sCH133cd9zgV29,1447197624013.c2a9f77791512281e4491e0797e824e8.
>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>> 2015-11-10 22:31:16,259 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,259 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,259 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,259 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,262 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,262 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,265 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,265 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,265 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,265 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,267 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,268 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,272 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,273 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,274 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>> 1fbbd37bdff125be32d352c98dffd2b1; next sequenceid=180083
>> 2015-11-10 22:31:16,274 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>> 7ce823a4e0f56bf8c854175409bb17b6; next sequenceid=21535
>> 2015-11-10 22:31:16,275 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>> for
>> prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1.
>> 2015-11-10 22:31:16,276 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>> for
>> prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6.
>> 2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>> Updated row
>> prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1.
>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>> 2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>> Updated row
>> prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6.
>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>> 2015-11-10 22:31:16,282 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,282 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,282 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>> 15701b8131ca505f6cce148d49cfbeee; next sequenceid=1029
>> 2015-11-10 22:31:16,283 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,283 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,284 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>> for
>> prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee.
>> 2015-11-10 22:31:16,304 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>> Updated row
>> prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee.
>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>> 2015-11-10 22:31:16,308 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,308 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,308 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,309 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,318 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>> 21b24baeaf26e12d02f79f243076850a; next sequenceid=114628
>> 2015-11-10 22:31:16,318 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>> cc52e0ba23708df10d696d0e33d9d137; next sequenceid=161394
>> 2015-11-10 22:31:16,320 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>> for
>> prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a.
>> 2015-11-10 22:31:16,320 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>> for prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD|2014111212
>> ,1447198690089.cc52e0ba23708df10d696d0e33d9d137.
>> 2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>> Updated row prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD|
>> 2014111212,1447198690089.cc52e0ba23708df10d696d0e33d9d137. with server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>> 2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>> Updated row
>> prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a.
>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>> 2015-11-10 22:31:21,184 WARN org.apache.hadoop.metrics2.lib.Interns:
>> Metrics intern cache overflow at 2011 for
>> RegionServer={RegionServer=MetricsInfoImpl{name=RegionServer,
>> description=RegionServer}, RegionServer
>> record=MetricsInfoImpl{name=RegionServer, description=RegionServer record}}
>> 2015-11-10 22:31:22,377 WARN
>> org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil:
>> hbase.regionserver.global.memstore.upperLimit is deprecated by
>> hbase.regionserver.global.memstore.size
>> 2015-11-10 22:31:22,403 INFO org.apache.hadoop.hbase.regionserver.HStore:
>> Completed compaction of 3 (all) file(s) in f of
>> prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041.
>> into d98861e6b5364d819d11c3f3e1c8fb56(size=180.4 M), total size for store
>> is 180.4 M. This selection was in queue for 0sec, and took 7sec to execute.
>> 2015-11-10 22:31:22,405 INFO
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>> compaction: Request =
>> regionName=prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041.,
>> storeName=f, fileCount=3, fileSize=180.4 M, priority=27,
>> time=741524299592168; duration=7sec
>> 2015-11-10 22:31:22,413 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on f in
>> region
>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
>> 2015-11-10 22:31:22,413 INFO org.apache.hadoop.hbase.regionserver.HStore:
>> Starting compaction of 3 file(s) in f of
>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
>> into tmpdir=hdfs://
>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_transactions/72f5a6c7dba1e486eae6b75f4e9f0d5d/.tmp,
>> totalSize=82.2 M
>> 2015-11-10 22:31:22,416 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:25,301 INFO org.apache.hadoop.hbase.regionserver.HStore:
>> Completed compaction of 3 (all) file(s) in f of
>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
>> into f7678de8dd8149428938a0aedfcbcaf9(size=82.2 M), total size for store is
>> 82.2 M. This selection was in queue for 0sec, and took 2sec to execute.
>> 2015-11-10 22:31:25,302 INFO
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>> compaction: Request =
>> regionName=prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.,
>> storeName=f, fileCount=3, fileSize=82.2 M, priority=27,
>> time=741532147583369; duration=2sec
>> 2015-11-10 22:31:25,305 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on f in
>> region
>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
>> 2015-11-10 22:31:25,305 INFO org.apache.hadoop.hbase.regionserver.HStore:
>> Starting compaction of 3 file(s) in f of
>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
>> into tmpdir=hdfs://
>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_offers/05284b692219df79852f3deaa2ce1927/.tmp,
>> totalSize=218.4 M
>> 2015-11-10 22:31:25,310 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:39,481 INFO org.apache.hadoop.hbase.regionserver.HStore:
>> Completed compaction of 3 (all) file(s) in f of
>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
>> into 4aea6e4301f848ff9bf727f4e511978a(size=214.6 M), total size for store
>> is 214.6 M. This selection was in queue for 0sec, and took 14sec to execute.
>> 2015-11-10 22:31:39,481 INFO
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>> compaction: Request =
>> regionName=prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.,
>> storeName=f, fileCount=3, fileSize=218.4 M, priority=27,
>> time=741535039619615; duration=14sec
>> 2015-11-10 22:31:39,484 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on d in
>> region
>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
>> 2015-11-10 22:31:39,484 INFO org.apache.hadoop.hbase.regionserver.HStore:
>> Starting compaction of 3 file(s) in d of
>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
>> into tmpdir=hdfs://
>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_balance_changes/40d39088cc3efe2d230a7d6f8d7b205b/.tmp,
>> totalSize=197.7 M
>> 2015-11-10 22:31:39,490 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:52,393 INFO org.apache.hadoop.hbase.regionserver.HStore:
>> Completed compaction of 3 (all) file(s) in d of
>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
>> into 1e48a57cdb544e898e78c9e8276ec05b(size=197.7 M), total size for store
>> is 197.7 M. This selection was in queue for 0sec, and took 12sec to execute.
>> 2015-11-10 22:31:52,393 INFO
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>> compaction: Request =
>> regionName=prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.,
>> storeName=d, fileCount=3, fileSize=197.7 M, priority=27,
>> time=741549219351203; duration=12sec
>> 2015-11-10 22:32:01,402 INFO
>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Atomically
>> moving hadoop3-private.wdc01.infra.ripple.com,60020,1447102836904's wals
>> to my queue
>> 2015-11-10 22:32:01,405 WARN
>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer 1 didn't
>> exist, skipping the replay
>> 2015-11-10 22:32:01,406 WARN
>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer
>> 1-hadoop4-private.wdc01.infra.ripple.com,60020,1447102248964 didn't
>> exist, skipping the replay
>> 2015-11-10 22:32:01,412 WARN
>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Got exception
>> in copyQueuesFromRSUsingMulti:
>> org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode =
>> Directory not empty
>> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
>> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1152)
>> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:1118)
>> at
>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:685)
>> at
>> org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1550)
>> at
>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.copyQueuesFromRSUsingMulti(ReplicationQueuesZKImpl.java:340)
>> at
>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.claimQueues(ReplicationQueuesZKImpl.java:172)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:570)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2015-11-10 22:36:01,223 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=3.78 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=4, accesses=63330, hits=6,
>> hitRatio=0.01%, , cachingAccesses=21, cachingHits=4,
>> cachingHitsRatio=19.05%, evictions=29, evicted=3,
>> evictedPerRun=0.1034482792019844
>> 2015-11-10 22:41:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.15 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=7, accesses=63856, hits=8,
>> hitRatio=0.01%, , cachingAccesses=32, cachingHits=6,
>> cachingHitsRatio=18.75%, evictions=59, evicted=3,
>> evictedPerRun=0.050847455859184265
>> 2015-11-10 22:46:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
>> cachingHitsRatio=21.43%, evictions=89, evicted=3,
>> evictedPerRun=0.033707864582538605
>> 2015-11-10 22:51:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
>> cachingHitsRatio=21.43%, evictions=119, evicted=3,
>> evictedPerRun=0.02521008439362049
>> 2015-11-10 22:56:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
>> cachingHitsRatio=21.43%, evictions=149, evicted=3,
>> evictedPerRun=0.020134227350354195
>> 2015-11-10 23:01:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63872, hits=13,
>> hitRatio=0.02%, , cachingAccesses=48, cachingHits=11,
>> cachingHitsRatio=22.92%, evictions=179, evicted=3,
>> evictedPerRun=0.016759777441620827
>> 2015-11-10 23:06:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63890, hits=21,
>> hitRatio=0.03%, , cachingAccesses=66, cachingHits=19,
>> cachingHitsRatio=28.79%, evictions=209, evicted=3,
>> evictedPerRun=0.014354066923260689
>> 2015-11-10 23:11:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64416, hits=26,
>> hitRatio=0.04%, , cachingAccesses=77, cachingHits=24,
>> cachingHitsRatio=31.17%, evictions=239, evicted=3,
>> evictedPerRun=0.012552301399409771
>> 2015-11-10 23:16:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
>> cachingHitsRatio=33.33%, evictions=269, evicted=3,
>> evictedPerRun=0.011152416467666626
>> 2015-11-10 23:21:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
>> cachingHitsRatio=33.33%, evictions=299, evicted=3,
>> evictedPerRun=0.010033444501459599
>> 2015-11-10 23:26:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
>> cachingHitsRatio=33.33%, evictions=329, evicted=3,
>> evictedPerRun=0.009118541143834591
>> 2015-11-10 23:31:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64432, hits=34,
>> hitRatio=0.05%, , cachingAccesses=93, cachingHits=32,
>> cachingHitsRatio=34.41%, evictions=359, evicted=3,
>> evictedPerRun=0.008356546051800251
>> 2015-11-10 23:31:01,228 INFO org.apache.hadoop.hbase.mob.MobFileCache:
>> MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted
>> files: 0
>> 2015-11-10 23:31:06,790 INFO
>> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Rolled WAL /hbase/WALs/
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/
>> hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
>> with entries=247, filesize=125.29 KB; new WAL /hbase/WALs/
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/
>> hadoop4-private.wdc01.infra.ripple.com
>> %2C60020%2C1447216261115.null0.1447219866745
>> 2015-11-10 23:31:06,792 INFO
>> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Archiving hdfs://
>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/WALs/hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
>> to hdfs://
>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/oldWALs/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
>> 2015-11-10 23:36:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64450, hits=42,
>> hitRatio=0.07%, , cachingAccesses=111, cachingHits=40,
>> cachingHitsRatio=36.04%, evictions=389, evicted=3,
>> evictedPerRun=0.007712082471698523
>> 2015-11-10 23:41:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64976, hits=47,
>> hitRatio=0.07%, , cachingAccesses=122, cachingHits=45,
>> cachingHitsRatio=36.89%, evictions=419, evicted=3,
>> evictedPerRun=0.0071599045768380165
>> 2015-11-10 23:46:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
>> cachingHitsRatio=37.88%, evictions=449, evicted=3,
>> evictedPerRun=0.00668151443824172
>> 2015-11-10 23:51:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
>> cachingHitsRatio=37.88%, evictions=479, evicted=3,
>> evictedPerRun=0.00626304792240262
>> 2015-11-10 23:56:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
>> cachingHitsRatio=37.88%, evictions=509, evicted=3,
>> evictedPerRun=0.005893909838050604
>> 2015-11-11 00:01:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64992, hits=55,
>> hitRatio=0.08%, , cachingAccesses=138, cachingHits=53,
>> cachingHitsRatio=38.41%, evictions=539, evicted=3,
>> evictedPerRun=0.005565862637013197
>> 2015-11-11 00:06:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65010, hits=63,
>> hitRatio=0.10%, , cachingAccesses=156, cachingHits=61,
>> cachingHitsRatio=39.10%, evictions=569, evicted=3,
>> evictedPerRun=0.005272407550364733
>> 2015-11-11 00:11:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65536, hits=68,
>> hitRatio=0.10%, , cachingAccesses=167, cachingHits=66,
>> cachingHitsRatio=39.52%, evictions=599, evicted=3,
>> evictedPerRun=0.00500834733247757
>> 2015-11-11 00:16:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65546, hits=73,
>> hitRatio=0.11%, , cachingAccesses=177, cachingHits=71,
>> cachingHitsRatio=40.11%, evictions=629, evicted=3,
>> evictedPerRun=0.004769475199282169
>>
>>
>>
>>
>> On Tue, Nov 10, 2015 at 8:15 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> bq. The region exits are forcing the data to start transferring to
>>> other regions,
>>>
>>> Can you elaborate a bit more on the above ? Did you mean some regions
>>> were transferred to other region server(s) ?
>>>
>>> Can you determined the effective RegionSplitPolicy ?
>>>
>>> Snippets of server logs would help illustrate the situation.
>>>
>>> Thanks
>>>
>>> On Tue, Nov 10, 2015 at 8:09 PM, Abraham Tom <wo...@gmail.com>
>>> wrote:
>>>
>>>> I am on CDH 5.4.8 hbase 1.0
>>>>
>>>> I am migrating to a new cluster and the data is sourced from CDH 5.3
>>>> hbase
>>>> 0.98.6
>>>>
>>>> I took snapshots from the old cluster exported them to the new cluster
>>>> - no
>>>> errors
>>>>
>>>> cloned the table, flush, major_compact and balanced the regions
>>>>
>>>> the configs are practically identical to the old cluster
>>>>
>>>>
>>>>
>>>> hbck and fsck all check out with no inconsistencies found
>>>>
>>>>
>>>>
>>>> yet when we try to load data via thrift the regions start splitting at
>>>> an
>>>> outrageous pace.
>>>>
>>>> Eventually the regions become offline starting a snowball affect. The
>>>> region exits are forcing the data to start transferring to other
>>>> regions,
>>>> with that creates large GC activity and multiple regions in transition
>>>> before thrift can no longer execute any writes. This usually happens
>>>> after
>>>> a few records. How we put is that we perform rowkey scans before puts
>>>> to
>>>> ensure where we last left off.
>>>>
>>>>
>>>>
>>>> Thrift eventually stops writing due to regions being offline and no
>>>> longer
>>>> serving. The regions eventually stop splitting and all the GC
>>>> eventually
>>>> complete. Once things settle back down, I am able to perform hbck and
>>>> fsck
>>>> and it again comes back with no inconsistencies
>>>>
>>>>
>>>>
>>>> I need ideas as to what is causing this
>>>>
>>>>
>>>> --
>>>> Abraham Tom
>>>> Email: work2much@gmail.com
>>>> Phone: 415-515-3621
>>>>
>>>
>>>
>>
>>
>> --
>> Abraham Tom
>> Email: work2much@gmail.com
>> Phone: 415-515-3621
>>
>
>
--
Abraham Tom
Email: work2much@gmail.com
Phone: 415-515-3621
Re: hbase regions behaving badly
Posted by Ted Yu <yu...@gmail.com>.
Please note that IncreasingToUpperBoundRegionSplitPolicy was in effect.
See http://hbase.apache.org/book.html#_custom_split_policies for related
information (there is link to
http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.html
).
In the future, please consider using pastebin etc. for server log so that
the message is concise.
Cheers
On Tue, Nov 10, 2015 at 10:17 PM, Abraham Tom <wo...@gmail.com> wrote:
> my hbase-site.xml snippet
>
>
> <property><name>hbase.mob.sweep.tool.compaction.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
>
> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
>
> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value><source>hbase-default.xml</source></property>
>
> <property><name>hbase.mob.sweep.tool.compaction.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
>
> <property><name>dfs.blockreport.split.threshold</name><value>1000000</value><source>hdfs-default.xml</source></property>
>
> <property><name>hbase.regionserver.regionSplitLimit</name><value>1000</value><source>hbase-default.xml</source></property>
>
> <property><name>hbase.splitlog.manager.timeout</name><value>120000</value><source>hbase-site.xml</source></property>
>
> <property><name>hbase.regionserver.region.split.policy</name><value>org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy</value><source>hbase-default.xml</source></property>
>
> <property><name>mapreduce.job.split.metainfo.maxsize</name><value>10000000</value><source>mapred-default.xml</source></property>
>
>
>
>
>
> and some master logs
> tail -100
> hbase-cmf-hbase-MASTER-hadoop4-private.wdc01.infra.ripple.com.log.out
> 2015-11-10 22:31:16,205 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {5097e5a15891c375eca53b992584c52e state=PENDING_OPEN,
> ts=1447216273284, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> to {5097e5a15891c375eca53b992584c52e state=OPENING, ts=1447216276205,
> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> 2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined d3b39413ee1bc88a100cd96564c4c420 from
> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
> 2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {cff8a38c23c3987fbbebec615f45f4ac state=OPENING,
> ts=1447216276185, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> to {cff8a38c23c3987fbbebec615f45f4ac state=OPEN, ts=1447216276206, server=
> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> 2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined bc842e97c8ff7b0f3498b38510a9c1c1 from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {c73147aef08804451c26c1f8099008f2 state=PENDING_OPEN,
> ts=1447216273284, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> to {c73147aef08804451c26c1f8099008f2 state=OPENING, ts=1447216276206,
> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> 2015-11-10 22:31:16,207 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined cff8a38c23c3987fbbebec615f45f4ac from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,208 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {8a671fb436fb0d186b37211d847aa897 state=OPENING,
> ts=1447216276188, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {8a671fb436fb0d186b37211d847aa897 state=OPEN, ts=1447216276208, server=
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,209 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined 8a671fb436fb0d186b37211d847aa897 from
> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
> 2015-11-10 22:31:16,209 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {e920fab780cdd6ba183bc50a8b5d8c72 state=PENDING_OPEN,
> ts=1447216273276, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING, ts=1447216276209,
> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,210 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {1258fdeaf4f00989a8282e5f9c215d7e state=OPENING,
> ts=1447216276193, server=hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
> to {1258fdeaf4f00989a8282e5f9c215d7e state=OPEN, ts=1447216276210, server=
> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
> 2015-11-10 22:31:16,211 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined 1258fdeaf4f00989a8282e5f9c215d7e from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,213 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {02666182c37df5043acdcaca31ceb095 state=OPENING,
> ts=1447216276184, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> to {02666182c37df5043acdcaca31ceb095 state=OPEN, ts=1447216276213, server=
> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> 2015-11-10 22:31:16,214 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {dae9fb9b19a8d2076fd4be2b95aefe45 state=PENDING_OPEN,
> ts=1447216273284, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> to {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING, ts=1447216276214,
> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> 2015-11-10 22:31:16,215 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined 02666182c37df5043acdcaca31ceb095 from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,216 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {4c694bcbe9679747e738a630aec33c3c state=OPENING,
> ts=1447216276200, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {4c694bcbe9679747e738a630aec33c3c state=OPEN, ts=1447216276216, server=
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,217 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined 4c694bcbe9679747e738a630aec33c3c from
> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
> 2015-11-10 22:31:16,217 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {dc3169f842a4d19014f040c887b9acff state=PENDING_OPEN,
> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {dc3169f842a4d19014f040c887b9acff state=OPENING, ts=1447216276217,
> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,219 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {8320a058823353a19f69a10da6827953 state=OPENING,
> ts=1447216276198, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {8320a058823353a19f69a10da6827953 state=OPEN, ts=1447216276219, server=
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,219 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined 8320a058823353a19f69a10da6827953 from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,220 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {edfd9220fe6c2a37fe7521101d490941 state=PENDING_OPEN,
> ts=1447216273142, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {edfd9220fe6c2a37fe7521101d490941 state=OPENING, ts=1447216276220,
> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,220 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {1937fd49e94d42fe072a29ff7af885bb state=OPENING,
> ts=1447216276205, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> to {1937fd49e94d42fe072a29ff7af885bb state=OPEN, ts=1447216276220, server=
> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> 2015-11-10 22:31:16,220 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {e4b861ca490c05fa4276e73e629b471a state=OPENING,
> ts=1447216276204, server=hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
> to {e4b861ca490c05fa4276e73e629b471a state=OPEN, ts=1447216276220, server=
> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
> 2015-11-10 22:31:16,221 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined 1937fd49e94d42fe072a29ff7af885bb from
> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
> 2015-11-10 22:31:16,221 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {9ecf375c160ffe4616b8c4fb45ea8fba state=PENDING_OPEN,
> ts=1447216273274, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> to {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING, ts=1447216276221,
> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> 2015-11-10 22:31:16,221 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined e4b861ca490c05fa4276e73e629b471a from
> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
> 2015-11-10 22:31:16,222 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {2c7572eeff0a72eceec10e0301d344a7 state=OPENING,
> ts=1447216276205, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> to {2c7572eeff0a72eceec10e0301d344a7 state=OPEN, ts=1447216276222, server=
> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> 2015-11-10 22:31:16,222 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined 2c7572eeff0a72eceec10e0301d344a7 from
> hadoop3-private.wdc01.infra.ripple.com,60020,1447208454196
> 2015-11-10 22:31:16,223 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {cb7e639dbb58b1e47c1da538165c503c state=PENDING_OPEN,
> ts=1447216273288, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> to {cb7e639dbb58b1e47c1da538165c503c state=OPENING, ts=1447216276223,
> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> 2015-11-10 22:31:16,225 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {5097e5a15891c375eca53b992584c52e state=OPENING,
> ts=1447216276205, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> to {5097e5a15891c375eca53b992584c52e state=OPEN, ts=1447216276225, server=
> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> 2015-11-10 22:31:16,225 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {c73147aef08804451c26c1f8099008f2 state=OPENING,
> ts=1447216276206, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> to {c73147aef08804451c26c1f8099008f2 state=OPEN, ts=1447216276225, server=
> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> 2015-11-10 22:31:16,226 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {abddf225d895bd30e11205390fedc03e state=OPENING,
> ts=1447216276205, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> to {abddf225d895bd30e11205390fedc03e state=OPEN, ts=1447216276226, server=
> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> 2015-11-10 22:31:16,226 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined 5097e5a15891c375eca53b992584c52e from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,226 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined c73147aef08804451c26c1f8099008f2 from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,227 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined abddf225d895bd30e11205390fedc03e from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,227 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {a5e6796ae343481b92b0180d817c1e17 state=PENDING_OPEN,
> ts=1447216273288, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> to {a5e6796ae343481b92b0180d817c1e17 state=OPENING, ts=1447216276227,
> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> 2015-11-10 22:31:16,229 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING,
> ts=1447216276214, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> to {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPEN, ts=1447216276229, server=
> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
> 2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING,
> ts=1447216276209, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {e920fab780cdd6ba183bc50a8b5d8c72 state=OPEN, ts=1447216276230, server=
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined dae9fb9b19a8d2076fd4be2b95aefe45 from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined e920fab780cdd6ba183bc50a8b5d8c72 from
> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
> 2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {2c04351380cc266df609ab53ef0e56ad state=PENDING_OPEN,
> ts=1447216273276, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {2c04351380cc266df609ab53ef0e56ad state=OPENING, ts=1447216276230,
> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,236 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {edfd9220fe6c2a37fe7521101d490941 state=OPENING,
> ts=1447216276220, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {edfd9220fe6c2a37fe7521101d490941 state=OPEN, ts=1447216276236, server=
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,236 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING,
> ts=1447216276221, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> to {9ecf375c160ffe4616b8c4fb45ea8fba state=OPEN, ts=1447216276236, server=
> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> 2015-11-10 22:31:16,237 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {dc3169f842a4d19014f040c887b9acff state=OPENING,
> ts=1447216276217, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {dc3169f842a4d19014f040c887b9acff state=OPEN, ts=1447216276237, server=
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,240 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined edfd9220fe6c2a37fe7521101d490941 from
> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
> 2015-11-10 22:31:16,240 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {a4f70c6c01f31f6d3d65e0cf55200c80 state=PENDING_OPEN,
> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING, ts=1447216276240,
> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,241 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {c2a9f77791512281e4491e0797e824e8 state=PENDING_OPEN,
> ts=1447216273277, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {c2a9f77791512281e4491e0797e824e8 state=OPENING, ts=1447216276241,
> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,241 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined dc3169f842a4d19014f040c887b9acff from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,241 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined 9ecf375c160ffe4616b8c4fb45ea8fba from
> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
> 2015-11-10 22:31:16,244 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {cb7e639dbb58b1e47c1da538165c503c state=OPENING,
> ts=1447216276223, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> to {cb7e639dbb58b1e47c1da538165c503c state=OPEN, ts=1447216276244, server=
> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> 2015-11-10 22:31:16,245 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined cb7e639dbb58b1e47c1da538165c503c from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,246 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {a5e6796ae343481b92b0180d817c1e17 state=OPENING,
> ts=1447216276227, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> to {a5e6796ae343481b92b0180d817c1e17 state=OPEN, ts=1447216276246, server=
> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
> 2015-11-10 22:31:16,246 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined a5e6796ae343481b92b0180d817c1e17 from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,252 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {2c04351380cc266df609ab53ef0e56ad state=OPENING,
> ts=1447216276230, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {2c04351380cc266df609ab53ef0e56ad state=OPEN, ts=1447216276252, server=
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,253 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined 2c04351380cc266df609ab53ef0e56ad from
> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
> 2015-11-10 22:31:16,253 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {15701b8131ca505f6cce148d49cfbeee state=PENDING_OPEN,
> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {15701b8131ca505f6cce148d49cfbeee state=OPENING, ts=1447216276253,
> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,257 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING,
> ts=1447216276240, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPEN, ts=1447216276257, server=
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,257 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {c2a9f77791512281e4491e0797e824e8 state=OPENING,
> ts=1447216276241, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {c2a9f77791512281e4491e0797e824e8 state=OPEN, ts=1447216276257, server=
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined a4f70c6c01f31f6d3d65e0cf55200c80 from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined c2a9f77791512281e4491e0797e824e8 from
> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
> 2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {7ce823a4e0f56bf8c854175409bb17b6 state=PENDING_OPEN,
> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING, ts=1447216276258,
> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {1fbbd37bdff125be32d352c98dffd2b1 state=PENDING_OPEN,
> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING, ts=1447216276258,
> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,280 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING,
> ts=1447216276258, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {1fbbd37bdff125be32d352c98dffd2b1 state=OPEN, ts=1447216276280, server=
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,280 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING,
> ts=1447216276258, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {7ce823a4e0f56bf8c854175409bb17b6 state=OPEN, ts=1447216276280, server=
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined 1fbbd37bdff125be32d352c98dffd2b1 from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined 7ce823a4e0f56bf8c854175409bb17b6 from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {cc52e0ba23708df10d696d0e33d9d137 state=PENDING_OPEN,
> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {cc52e0ba23708df10d696d0e33d9d137 state=OPENING, ts=1447216276281,
> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {21b24baeaf26e12d02f79f243076850a state=PENDING_OPEN,
> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {21b24baeaf26e12d02f79f243076850a state=OPENING, ts=1447216276281,
> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,306 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {15701b8131ca505f6cce148d49cfbeee state=OPENING,
> ts=1447216276253, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {15701b8131ca505f6cce148d49cfbeee state=OPEN, ts=1447216276306, server=
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,307 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined 15701b8131ca505f6cce148d49cfbeee from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,325 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {cc52e0ba23708df10d696d0e33d9d137 state=OPENING,
> ts=1447216276281, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {cc52e0ba23708df10d696d0e33d9d137 state=OPEN, ts=1447216276325, server=
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,325 INFO org.apache.hadoop.hbase.master.RegionStates:
> Transition {21b24baeaf26e12d02f79f243076850a state=OPENING,
> ts=1447216276281, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> to {21b24baeaf26e12d02f79f243076850a state=OPEN, ts=1447216276325, server=
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
> 2015-11-10 22:31:16,326 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined cc52e0ba23708df10d696d0e33d9d137 from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:31:16,326 INFO org.apache.hadoop.hbase.master.RegionStates:
> Offlined 21b24baeaf26e12d02f79f243076850a from
> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
> 2015-11-10 22:36:01,090 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0
> 2015-11-10 22:41:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0
> 2015-11-10 22:46:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=89, evicted=0, evictedPerRun=0.0
> 2015-11-10 22:47:34,680 INFO
> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
> 10.125.122.237 set balanceSwitch=false
> 2015-11-10 22:47:41,845 INFO
> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
> 10.125.122.237 set balanceSwitch=true
> 2015-11-10 22:51:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
> 2015-11-10 22:56:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=149, evicted=0, evictedPerRun=0.0
> 2015-11-10 23:01:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=179, evicted=0, evictedPerRun=0.0
> 2015-11-10 23:06:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=209, evicted=0, evictedPerRun=0.0
> 2015-11-10 23:11:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=239, evicted=0, evictedPerRun=0.0
> 2015-11-10 23:16:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=269, evicted=0, evictedPerRun=0.0
> 2015-11-10 23:17:34,662 INFO
> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
> 10.125.122.237 set balanceSwitch=false
> 2015-11-10 23:17:38,707 INFO
> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
> 10.125.122.237 set balanceSwitch=true
> 2015-11-10 23:21:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=299, evicted=0, evictedPerRun=0.0
> 2015-11-10 23:26:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=329, evicted=0, evictedPerRun=0.0
> 2015-11-10 23:31:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=359, evicted=0, evictedPerRun=0.0
> 2015-11-10 23:31:01,096 INFO org.apache.hadoop.hbase.mob.MobFileCache:
> MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted
> files: 0
> 2015-11-10 23:36:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=389, evicted=0, evictedPerRun=0.0
> 2015-11-10 23:41:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=419, evicted=0, evictedPerRun=0.0
> 2015-11-10 23:46:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=449, evicted=0, evictedPerRun=0.0
> 2015-11-10 23:47:34,668 INFO
> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
> 10.125.122.237 set balanceSwitch=false
> 2015-11-10 23:47:44,843 INFO
> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
> 10.125.122.237 set balanceSwitch=true
> 2015-11-10 23:51:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=479, evicted=0, evictedPerRun=0.0
> 2015-11-10 23:56:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=509, evicted=0, evictedPerRun=0.0
> 2015-11-11 00:01:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=539, evicted=0, evictedPerRun=0.0
> 2015-11-11 00:06:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=569, evicted=0, evictedPerRun=0.0
> 2015-11-11 00:11:01,089 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
> hitRatio=0, cachingAccesses=0, cachingHits=0,
> cachingHitsRatio=0,evictions=599, evicted=0, evictedPerRun=0.0
>
> Region server logs
> tail -100
> hbase-cmf-hbase-REGIONSERVER-hadoop4-private.wdc01.infra.ripple.com.log.out
> 2015-11-10 22:31:16,255 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2015-11-10 22:31:16,255 INFO
> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
> major jitter 0.500000, min locality to compact 0.000000
> 2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor:
> Updated row prod_control,,1434077692105.a4f70c6c01f31f6d3d65e0cf55200c80.
> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
> 2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor:
> Updated row
> prod_agg_account_payments,20140325000000|rH48e8bf5kNWuFRBwgm5sCH133cd9zgV29,1447197624013.c2a9f77791512281e4491e0797e824e8.
> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
> 2015-11-10 22:31:16,259 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2015-11-10 22:31:16,259 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2015-11-10 22:31:16,259 INFO
> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
> major jitter 0.500000, min locality to compact 0.000000
> 2015-11-10 22:31:16,259 INFO
> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
> major jitter 0.500000, min locality to compact 0.000000
> 2015-11-10 22:31:16,262 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2015-11-10 22:31:16,262 INFO
> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
> major jitter 0.500000, min locality to compact 0.000000
> 2015-11-10 22:31:16,265 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2015-11-10 22:31:16,265 INFO
> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
> major jitter 0.500000, min locality to compact 0.000000
> 2015-11-10 22:31:16,265 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2015-11-10 22:31:16,265 INFO
> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
> major jitter 0.500000, min locality to compact 0.000000
> 2015-11-10 22:31:16,267 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2015-11-10 22:31:16,268 INFO
> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
> major jitter 0.500000, min locality to compact 0.000000
> 2015-11-10 22:31:16,272 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2015-11-10 22:31:16,273 INFO
> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
> major jitter 0.500000, min locality to compact 0.000000
> 2015-11-10 22:31:16,274 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Onlined 1fbbd37bdff125be32d352c98dffd2b1; next sequenceid=180083
> 2015-11-10 22:31:16,274 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Onlined 7ce823a4e0f56bf8c854175409bb17b6; next sequenceid=21535
> 2015-11-10 22:31:16,275 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
> for
> prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1.
> 2015-11-10 22:31:16,276 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
> for
> prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6.
> 2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor:
> Updated row
> prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1.
> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
> 2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor:
> Updated row
> prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6.
> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
> 2015-11-10 22:31:16,282 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2015-11-10 22:31:16,282 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2015-11-10 22:31:16,282 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Onlined 15701b8131ca505f6cce148d49cfbeee; next sequenceid=1029
> 2015-11-10 22:31:16,283 INFO
> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
> major jitter 0.500000, min locality to compact 0.000000
> 2015-11-10 22:31:16,283 INFO
> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
> major jitter 0.500000, min locality to compact 0.000000
> 2015-11-10 22:31:16,284 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
> for
> prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee.
> 2015-11-10 22:31:16,304 INFO org.apache.hadoop.hbase.MetaTableAccessor:
> Updated row
> prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee.
> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
> 2015-11-10 22:31:16,308 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2015-11-10 22:31:16,308 INFO
> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
> major jitter 0.500000, min locality to compact 0.000000
> 2015-11-10 22:31:16,308 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2015-11-10 22:31:16,309 INFO
> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
> major jitter 0.500000, min locality to compact 0.000000
> 2015-11-10 22:31:16,318 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Onlined 21b24baeaf26e12d02f79f243076850a; next sequenceid=114628
> 2015-11-10 22:31:16,318 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Onlined cc52e0ba23708df10d696d0e33d9d137; next sequenceid=161394
> 2015-11-10 22:31:16,320 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
> for
> prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a.
> 2015-11-10 22:31:16,320 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
> for prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD|2014111212
> ,1447198690089.cc52e0ba23708df10d696d0e33d9d137.
> 2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor:
> Updated row prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD|
> 2014111212,1447198690089.cc52e0ba23708df10d696d0e33d9d137. with server=
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
> 2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor:
> Updated row
> prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a.
> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
> 2015-11-10 22:31:21,184 WARN org.apache.hadoop.metrics2.lib.Interns:
> Metrics intern cache overflow at 2011 for
> RegionServer={RegionServer=MetricsInfoImpl{name=RegionServer,
> description=RegionServer}, RegionServer
> record=MetricsInfoImpl{name=RegionServer, description=RegionServer record}}
> 2015-11-10 22:31:22,377 WARN
> org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil:
> hbase.regionserver.global.memstore.upperLimit is deprecated by
> hbase.regionserver.global.memstore.size
> 2015-11-10 22:31:22,403 INFO org.apache.hadoop.hbase.regionserver.HStore:
> Completed compaction of 3 (all) file(s) in f of
> prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041.
> into d98861e6b5364d819d11c3f3e1c8fb56(size=180.4 M), total size for store
> is 180.4 M. This selection was in queue for 0sec, and took 7sec to execute.
> 2015-11-10 22:31:22,405 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
> compaction: Request =
> regionName=prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041.,
> storeName=f, fileCount=3, fileSize=180.4 M, priority=27,
> time=741524299592168; duration=7sec
> 2015-11-10 22:31:22,413 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting compaction on f in region
> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
> 2015-11-10 22:31:22,413 INFO org.apache.hadoop.hbase.regionserver.HStore:
> Starting compaction of 3 file(s) in f of
> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
> into tmpdir=hdfs://
> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_transactions/72f5a6c7dba1e486eae6b75f4e9f0d5d/.tmp,
> totalSize=82.2 M
> 2015-11-10 22:31:22,416 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2015-11-10 22:31:25,301 INFO org.apache.hadoop.hbase.regionserver.HStore:
> Completed compaction of 3 (all) file(s) in f of
> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
> into f7678de8dd8149428938a0aedfcbcaf9(size=82.2 M), total size for store is
> 82.2 M. This selection was in queue for 0sec, and took 2sec to execute.
> 2015-11-10 22:31:25,302 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
> compaction: Request =
> regionName=prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.,
> storeName=f, fileCount=3, fileSize=82.2 M, priority=27,
> time=741532147583369; duration=2sec
> 2015-11-10 22:31:25,305 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting compaction on f in region
> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
> 2015-11-10 22:31:25,305 INFO org.apache.hadoop.hbase.regionserver.HStore:
> Starting compaction of 3 file(s) in f of
> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
> into tmpdir=hdfs://
> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_offers/05284b692219df79852f3deaa2ce1927/.tmp,
> totalSize=218.4 M
> 2015-11-10 22:31:25,310 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2015-11-10 22:31:39,481 INFO org.apache.hadoop.hbase.regionserver.HStore:
> Completed compaction of 3 (all) file(s) in f of
> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
> into 4aea6e4301f848ff9bf727f4e511978a(size=214.6 M), total size for store
> is 214.6 M. This selection was in queue for 0sec, and took 14sec to execute.
> 2015-11-10 22:31:39,481 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
> compaction: Request =
> regionName=prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.,
> storeName=f, fileCount=3, fileSize=218.4 M, priority=27,
> time=741535039619615; duration=14sec
> 2015-11-10 22:31:39,484 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Starting compaction on d in region
> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
> 2015-11-10 22:31:39,484 INFO org.apache.hadoop.hbase.regionserver.HStore:
> Starting compaction of 3 file(s) in d of
> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
> into tmpdir=hdfs://
> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_balance_changes/40d39088cc3efe2d230a7d6f8d7b205b/.tmp,
> totalSize=197.7 M
> 2015-11-10 22:31:39,490 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
> cacheDataCompressed=false, prefetchOnOpen=false
> 2015-11-10 22:31:52,393 INFO org.apache.hadoop.hbase.regionserver.HStore:
> Completed compaction of 3 (all) file(s) in d of
> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
> into 1e48a57cdb544e898e78c9e8276ec05b(size=197.7 M), total size for store
> is 197.7 M. This selection was in queue for 0sec, and took 12sec to execute.
> 2015-11-10 22:31:52,393 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
> compaction: Request =
> regionName=prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.,
> storeName=d, fileCount=3, fileSize=197.7 M, priority=27,
> time=741549219351203; duration=12sec
> 2015-11-10 22:32:01,402 INFO
> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Atomically
> moving hadoop3-private.wdc01.infra.ripple.com,60020,1447102836904's wals
> to my queue
> 2015-11-10 22:32:01,405 WARN
> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer 1 didn't
> exist, skipping the replay
> 2015-11-10 22:32:01,406 WARN
> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer
> 1-hadoop4-private.wdc01.infra.ripple.com,60020,1447102248964 didn't
> exist, skipping the replay
> 2015-11-10 22:32:01,412 WARN
> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Got exception
> in copyQueuesFromRSUsingMulti:
> org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode =
> Directory not empty
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1152)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:1118)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:685)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1550)
> at
> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.copyQueuesFromRSUsingMulti(ReplicationQueuesZKImpl.java:340)
> at
> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.claimQueues(ReplicationQueuesZKImpl.java:172)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:570)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2015-11-10 22:36:01,223 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=3.78 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=4, accesses=63330, hits=6,
> hitRatio=0.01%, , cachingAccesses=21, cachingHits=4,
> cachingHitsRatio=19.05%, evictions=29, evicted=3,
> evictedPerRun=0.1034482792019844
> 2015-11-10 22:41:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.15 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=7, accesses=63856, hits=8,
> hitRatio=0.01%, , cachingAccesses=32, cachingHits=6,
> cachingHitsRatio=18.75%, evictions=59, evicted=3,
> evictedPerRun=0.050847455859184265
> 2015-11-10 22:46:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
> cachingHitsRatio=21.43%, evictions=89, evicted=3,
> evictedPerRun=0.033707864582538605
> 2015-11-10 22:51:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
> cachingHitsRatio=21.43%, evictions=119, evicted=3,
> evictedPerRun=0.02521008439362049
> 2015-11-10 22:56:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
> cachingHitsRatio=21.43%, evictions=149, evicted=3,
> evictedPerRun=0.020134227350354195
> 2015-11-10 23:01:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63872, hits=13,
> hitRatio=0.02%, , cachingAccesses=48, cachingHits=11,
> cachingHitsRatio=22.92%, evictions=179, evicted=3,
> evictedPerRun=0.016759777441620827
> 2015-11-10 23:06:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63890, hits=21,
> hitRatio=0.03%, , cachingAccesses=66, cachingHits=19,
> cachingHitsRatio=28.79%, evictions=209, evicted=3,
> evictedPerRun=0.014354066923260689
> 2015-11-10 23:11:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64416, hits=26,
> hitRatio=0.04%, , cachingAccesses=77, cachingHits=24,
> cachingHitsRatio=31.17%, evictions=239, evicted=3,
> evictedPerRun=0.012552301399409771
> 2015-11-10 23:16:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
> cachingHitsRatio=33.33%, evictions=269, evicted=3,
> evictedPerRun=0.011152416467666626
> 2015-11-10 23:21:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
> cachingHitsRatio=33.33%, evictions=299, evicted=3,
> evictedPerRun=0.010033444501459599
> 2015-11-10 23:26:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
> cachingHitsRatio=33.33%, evictions=329, evicted=3,
> evictedPerRun=0.009118541143834591
> 2015-11-10 23:31:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64432, hits=34,
> hitRatio=0.05%, , cachingAccesses=93, cachingHits=32,
> cachingHitsRatio=34.41%, evictions=359, evicted=3,
> evictedPerRun=0.008356546051800251
> 2015-11-10 23:31:01,228 INFO org.apache.hadoop.hbase.mob.MobFileCache:
> MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted
> files: 0
> 2015-11-10 23:31:06,790 INFO
> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Rolled WAL /hbase/WALs/
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/
> hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
> with entries=247, filesize=125.29 KB; new WAL /hbase/WALs/
> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/
> hadoop4-private.wdc01.infra.ripple.com
> %2C60020%2C1447216261115.null0.1447219866745
> 2015-11-10 23:31:06,792 INFO
> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Archiving hdfs://
> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/WALs/hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
> to hdfs://
> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/oldWALs/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
> 2015-11-10 23:36:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64450, hits=42,
> hitRatio=0.07%, , cachingAccesses=111, cachingHits=40,
> cachingHitsRatio=36.04%, evictions=389, evicted=3,
> evictedPerRun=0.007712082471698523
> 2015-11-10 23:41:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64976, hits=47,
> hitRatio=0.07%, , cachingAccesses=122, cachingHits=45,
> cachingHitsRatio=36.89%, evictions=419, evicted=3,
> evictedPerRun=0.0071599045768380165
> 2015-11-10 23:46:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
> cachingHitsRatio=37.88%, evictions=449, evicted=3,
> evictedPerRun=0.00668151443824172
> 2015-11-10 23:51:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
> cachingHitsRatio=37.88%, evictions=479, evicted=3,
> evictedPerRun=0.00626304792240262
> 2015-11-10 23:56:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
> cachingHitsRatio=37.88%, evictions=509, evicted=3,
> evictedPerRun=0.005893909838050604
> 2015-11-11 00:01:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64992, hits=55,
> hitRatio=0.08%, , cachingAccesses=138, cachingHits=53,
> cachingHitsRatio=38.41%, evictions=539, evicted=3,
> evictedPerRun=0.005565862637013197
> 2015-11-11 00:06:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65010, hits=63,
> hitRatio=0.10%, , cachingAccesses=156, cachingHits=61,
> cachingHitsRatio=39.10%, evictions=569, evicted=3,
> evictedPerRun=0.005272407550364733
> 2015-11-11 00:11:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65536, hits=68,
> hitRatio=0.10%, , cachingAccesses=167, cachingHits=66,
> cachingHitsRatio=39.52%, evictions=599, evicted=3,
> evictedPerRun=0.00500834733247757
> 2015-11-11 00:16:01,222 INFO
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65546, hits=73,
> hitRatio=0.11%, , cachingAccesses=177, cachingHits=71,
> cachingHitsRatio=40.11%, evictions=629, evicted=3,
> evictedPerRun=0.004769475199282169
>
>
>
>
> On Tue, Nov 10, 2015 at 8:15 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> bq. The region exits are forcing the data to start transferring to
>> other regions,
>>
>> Can you elaborate a bit more on the above ? Did you mean some regions
>> were transferred to other region server(s) ?
>>
>> Can you determined the effective RegionSplitPolicy ?
>>
>> Snippets of server logs would help illustrate the situation.
>>
>> Thanks
>>
>> On Tue, Nov 10, 2015 at 8:09 PM, Abraham Tom <wo...@gmail.com> wrote:
>>
>>> I am on CDH 5.4.8 hbase 1.0
>>>
>>> I am migrating to a new cluster and the data is sourced from CDH 5.3
>>> hbase
>>> 0.98.6
>>>
>>> I took snapshots from the old cluster exported them to the new cluster -
>>> no
>>> errors
>>>
>>> cloned the table, flush, major_compact and balanced the regions
>>>
>>> the configs are practically identical to the old cluster
>>>
>>>
>>>
>>> hbck and fsck all check out with no inconsistencies found
>>>
>>>
>>>
>>> yet when we try to load data via thrift the regions start splitting at an
>>> outrageous pace.
>>>
>>> Eventually the regions become offline starting a snowball affect. The
>>> region exits are forcing the data to start transferring to other
>>> regions,
>>> with that creates large GC activity and multiple regions in transition
>>> before thrift can no longer execute any writes. This usually happens
>>> after
>>> a few records. How we put is that we perform rowkey scans before puts
>>> to
>>> ensure where we last left off.
>>>
>>>
>>>
>>> Thrift eventually stops writing due to regions being offline and no
>>> longer
>>> serving. The regions eventually stop splitting and all the GC eventually
>>> complete. Once things settle back down, I am able to perform hbck and
>>> fsck
>>> and it again comes back with no inconsistencies
>>>
>>>
>>>
>>> I need ideas as to what is causing this
>>>
>>>
>>> --
>>> Abraham Tom
>>> Email: work2much@gmail.com
>>> Phone: 415-515-3621
>>>
>>
>>
>
>
> --
> Abraham Tom
> Email: work2much@gmail.com
> Phone: 415-515-3621
>
Re: hbase regions behaving badly
Posted by Abraham Tom <wo...@gmail.com>.
my hbase-site.xml snippet
<property><name>hbase.mob.sweep.tool.compaction.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
<property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
<property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value><source>hbase-default.xml</source></property>
<property><name>hbase.mob.sweep.tool.compaction.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
<property><name>dfs.blockreport.split.threshold</name><value>1000000</value><source>hdfs-default.xml</source></property>
<property><name>hbase.regionserver.regionSplitLimit</name><value>1000</value><source>hbase-default.xml</source></property>
<property><name>hbase.splitlog.manager.timeout</name><value>120000</value><source>hbase-site.xml</source></property>
<property><name>hbase.regionserver.region.split.policy</name><value>org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy</value><source>hbase-default.xml</source></property>
<property><name>mapreduce.job.split.metainfo.maxsize</name><value>10000000</value><source>mapred-default.xml</source></property>
and some master logs
tail -100
hbase-cmf-hbase-MASTER-hadoop4-private.wdc01.infra.ripple.com.log.out
2015-11-10 22:31:16,205 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {5097e5a15891c375eca53b992584c52e state=PENDING_OPEN,
ts=1447216273284,
server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
to {5097e5a15891c375eca53b992584c52e state=OPENING, ts=1447216276205,
server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined d3b39413ee1bc88a100cd96564c4c420 from
hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {cff8a38c23c3987fbbebec615f45f4ac state=OPENING,
ts=1447216276185,
server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
to {cff8a38c23c3987fbbebec615f45f4ac state=OPEN, ts=1447216276206, server=
hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined bc842e97c8ff7b0f3498b38510a9c1c1 from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {c73147aef08804451c26c1f8099008f2 state=PENDING_OPEN,
ts=1447216273284,
server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
to {c73147aef08804451c26c1f8099008f2 state=OPENING, ts=1447216276206,
server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
2015-11-10 22:31:16,207 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined cff8a38c23c3987fbbebec615f45f4ac from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,208 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {8a671fb436fb0d186b37211d847aa897 state=OPENING,
ts=1447216276188,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {8a671fb436fb0d186b37211d847aa897 state=OPEN, ts=1447216276208, server=
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,209 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined 8a671fb436fb0d186b37211d847aa897 from
hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
2015-11-10 22:31:16,209 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {e920fab780cdd6ba183bc50a8b5d8c72 state=PENDING_OPEN,
ts=1447216273276,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING, ts=1447216276209,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,210 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {1258fdeaf4f00989a8282e5f9c215d7e state=OPENING,
ts=1447216276193,
server=hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
to {1258fdeaf4f00989a8282e5f9c215d7e state=OPEN, ts=1447216276210, server=
hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
2015-11-10 22:31:16,211 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined 1258fdeaf4f00989a8282e5f9c215d7e from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,213 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {02666182c37df5043acdcaca31ceb095 state=OPENING,
ts=1447216276184,
server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
to {02666182c37df5043acdcaca31ceb095 state=OPEN, ts=1447216276213, server=
hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
2015-11-10 22:31:16,214 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {dae9fb9b19a8d2076fd4be2b95aefe45 state=PENDING_OPEN,
ts=1447216273284,
server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
to {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING, ts=1447216276214,
server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
2015-11-10 22:31:16,215 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined 02666182c37df5043acdcaca31ceb095 from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,216 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {4c694bcbe9679747e738a630aec33c3c state=OPENING,
ts=1447216276200,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {4c694bcbe9679747e738a630aec33c3c state=OPEN, ts=1447216276216, server=
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,217 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined 4c694bcbe9679747e738a630aec33c3c from
hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
2015-11-10 22:31:16,217 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {dc3169f842a4d19014f040c887b9acff state=PENDING_OPEN,
ts=1447216273280,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {dc3169f842a4d19014f040c887b9acff state=OPENING, ts=1447216276217,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,219 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {8320a058823353a19f69a10da6827953 state=OPENING,
ts=1447216276198,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {8320a058823353a19f69a10da6827953 state=OPEN, ts=1447216276219, server=
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,219 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined 8320a058823353a19f69a10da6827953 from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,220 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {edfd9220fe6c2a37fe7521101d490941 state=PENDING_OPEN,
ts=1447216273142,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {edfd9220fe6c2a37fe7521101d490941 state=OPENING, ts=1447216276220,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,220 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {1937fd49e94d42fe072a29ff7af885bb state=OPENING,
ts=1447216276205,
server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
to {1937fd49e94d42fe072a29ff7af885bb state=OPEN, ts=1447216276220, server=
hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
2015-11-10 22:31:16,220 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {e4b861ca490c05fa4276e73e629b471a state=OPENING,
ts=1447216276204,
server=hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
to {e4b861ca490c05fa4276e73e629b471a state=OPEN, ts=1447216276220, server=
hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
2015-11-10 22:31:16,221 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined 1937fd49e94d42fe072a29ff7af885bb from
hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
2015-11-10 22:31:16,221 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {9ecf375c160ffe4616b8c4fb45ea8fba state=PENDING_OPEN,
ts=1447216273274,
server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
to {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING, ts=1447216276221,
server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
2015-11-10 22:31:16,221 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined e4b861ca490c05fa4276e73e629b471a from
hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
2015-11-10 22:31:16,222 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {2c7572eeff0a72eceec10e0301d344a7 state=OPENING,
ts=1447216276205,
server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
to {2c7572eeff0a72eceec10e0301d344a7 state=OPEN, ts=1447216276222, server=
hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
2015-11-10 22:31:16,222 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined 2c7572eeff0a72eceec10e0301d344a7 from
hadoop3-private.wdc01.infra.ripple.com,60020,1447208454196
2015-11-10 22:31:16,223 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {cb7e639dbb58b1e47c1da538165c503c state=PENDING_OPEN,
ts=1447216273288,
server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
to {cb7e639dbb58b1e47c1da538165c503c state=OPENING, ts=1447216276223,
server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
2015-11-10 22:31:16,225 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {5097e5a15891c375eca53b992584c52e state=OPENING,
ts=1447216276205,
server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
to {5097e5a15891c375eca53b992584c52e state=OPEN, ts=1447216276225, server=
hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
2015-11-10 22:31:16,225 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {c73147aef08804451c26c1f8099008f2 state=OPENING,
ts=1447216276206,
server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
to {c73147aef08804451c26c1f8099008f2 state=OPEN, ts=1447216276225, server=
hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
2015-11-10 22:31:16,226 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {abddf225d895bd30e11205390fedc03e state=OPENING,
ts=1447216276205,
server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
to {abddf225d895bd30e11205390fedc03e state=OPEN, ts=1447216276226, server=
hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
2015-11-10 22:31:16,226 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined 5097e5a15891c375eca53b992584c52e from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,226 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined c73147aef08804451c26c1f8099008f2 from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,227 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined abddf225d895bd30e11205390fedc03e from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,227 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {a5e6796ae343481b92b0180d817c1e17 state=PENDING_OPEN,
ts=1447216273288,
server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
to {a5e6796ae343481b92b0180d817c1e17 state=OPENING, ts=1447216276227,
server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
2015-11-10 22:31:16,229 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING,
ts=1447216276214,
server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
to {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPEN, ts=1447216276229, server=
hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING,
ts=1447216276209,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {e920fab780cdd6ba183bc50a8b5d8c72 state=OPEN, ts=1447216276230, server=
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined dae9fb9b19a8d2076fd4be2b95aefe45 from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined e920fab780cdd6ba183bc50a8b5d8c72 from
hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {2c04351380cc266df609ab53ef0e56ad state=PENDING_OPEN,
ts=1447216273276,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {2c04351380cc266df609ab53ef0e56ad state=OPENING, ts=1447216276230,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,236 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {edfd9220fe6c2a37fe7521101d490941 state=OPENING,
ts=1447216276220,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {edfd9220fe6c2a37fe7521101d490941 state=OPEN, ts=1447216276236, server=
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,236 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING,
ts=1447216276221,
server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
to {9ecf375c160ffe4616b8c4fb45ea8fba state=OPEN, ts=1447216276236, server=
hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
2015-11-10 22:31:16,237 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {dc3169f842a4d19014f040c887b9acff state=OPENING,
ts=1447216276217,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {dc3169f842a4d19014f040c887b9acff state=OPEN, ts=1447216276237, server=
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,240 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined edfd9220fe6c2a37fe7521101d490941 from
hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
2015-11-10 22:31:16,240 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {a4f70c6c01f31f6d3d65e0cf55200c80 state=PENDING_OPEN,
ts=1447216273280,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING, ts=1447216276240,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,241 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {c2a9f77791512281e4491e0797e824e8 state=PENDING_OPEN,
ts=1447216273277,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {c2a9f77791512281e4491e0797e824e8 state=OPENING, ts=1447216276241,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,241 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined dc3169f842a4d19014f040c887b9acff from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,241 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined 9ecf375c160ffe4616b8c4fb45ea8fba from
hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
2015-11-10 22:31:16,244 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {cb7e639dbb58b1e47c1da538165c503c state=OPENING,
ts=1447216276223,
server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
to {cb7e639dbb58b1e47c1da538165c503c state=OPEN, ts=1447216276244, server=
hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
2015-11-10 22:31:16,245 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined cb7e639dbb58b1e47c1da538165c503c from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,246 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {a5e6796ae343481b92b0180d817c1e17 state=OPENING,
ts=1447216276227,
server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
to {a5e6796ae343481b92b0180d817c1e17 state=OPEN, ts=1447216276246, server=
hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
2015-11-10 22:31:16,246 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined a5e6796ae343481b92b0180d817c1e17 from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,252 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {2c04351380cc266df609ab53ef0e56ad state=OPENING,
ts=1447216276230,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {2c04351380cc266df609ab53ef0e56ad state=OPEN, ts=1447216276252, server=
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,253 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined 2c04351380cc266df609ab53ef0e56ad from
hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
2015-11-10 22:31:16,253 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {15701b8131ca505f6cce148d49cfbeee state=PENDING_OPEN,
ts=1447216273280,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {15701b8131ca505f6cce148d49cfbeee state=OPENING, ts=1447216276253,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,257 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING,
ts=1447216276240,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPEN, ts=1447216276257, server=
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,257 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {c2a9f77791512281e4491e0797e824e8 state=OPENING,
ts=1447216276241,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {c2a9f77791512281e4491e0797e824e8 state=OPEN, ts=1447216276257, server=
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined a4f70c6c01f31f6d3d65e0cf55200c80 from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined c2a9f77791512281e4491e0797e824e8 from
hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {7ce823a4e0f56bf8c854175409bb17b6 state=PENDING_OPEN,
ts=1447216273280,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING, ts=1447216276258,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {1fbbd37bdff125be32d352c98dffd2b1 state=PENDING_OPEN,
ts=1447216273280,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING, ts=1447216276258,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,280 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING,
ts=1447216276258,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {1fbbd37bdff125be32d352c98dffd2b1 state=OPEN, ts=1447216276280, server=
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,280 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING,
ts=1447216276258,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {7ce823a4e0f56bf8c854175409bb17b6 state=OPEN, ts=1447216276280, server=
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined 1fbbd37bdff125be32d352c98dffd2b1 from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined 7ce823a4e0f56bf8c854175409bb17b6 from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {cc52e0ba23708df10d696d0e33d9d137 state=PENDING_OPEN,
ts=1447216273280,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {cc52e0ba23708df10d696d0e33d9d137 state=OPENING, ts=1447216276281,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {21b24baeaf26e12d02f79f243076850a state=PENDING_OPEN,
ts=1447216273280,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {21b24baeaf26e12d02f79f243076850a state=OPENING, ts=1447216276281,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,306 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {15701b8131ca505f6cce148d49cfbeee state=OPENING,
ts=1447216276253,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {15701b8131ca505f6cce148d49cfbeee state=OPEN, ts=1447216276306, server=
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,307 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined 15701b8131ca505f6cce148d49cfbeee from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,325 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {cc52e0ba23708df10d696d0e33d9d137 state=OPENING,
ts=1447216276281,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {cc52e0ba23708df10d696d0e33d9d137 state=OPEN, ts=1447216276325, server=
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,325 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {21b24baeaf26e12d02f79f243076850a state=OPENING,
ts=1447216276281,
server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
to {21b24baeaf26e12d02f79f243076850a state=OPEN, ts=1447216276325, server=
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
2015-11-10 22:31:16,326 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined cc52e0ba23708df10d696d0e33d9d137 from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:31:16,326 INFO org.apache.hadoop.hbase.master.RegionStates:
Offlined 21b24baeaf26e12d02f79f243076850a from
hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
2015-11-10 22:36:01,090 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0
2015-11-10 22:41:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0
2015-11-10 22:46:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=89, evicted=0, evictedPerRun=0.0
2015-11-10 22:47:34,680 INFO
org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
10.125.122.237 set balanceSwitch=false
2015-11-10 22:47:41,845 INFO
org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
10.125.122.237 set balanceSwitch=true
2015-11-10 22:51:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
2015-11-10 22:56:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=149, evicted=0, evictedPerRun=0.0
2015-11-10 23:01:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=179, evicted=0, evictedPerRun=0.0
2015-11-10 23:06:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=209, evicted=0, evictedPerRun=0.0
2015-11-10 23:11:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=239, evicted=0, evictedPerRun=0.0
2015-11-10 23:16:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=269, evicted=0, evictedPerRun=0.0
2015-11-10 23:17:34,662 INFO
org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
10.125.122.237 set balanceSwitch=false
2015-11-10 23:17:38,707 INFO
org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
10.125.122.237 set balanceSwitch=true
2015-11-10 23:21:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=299, evicted=0, evictedPerRun=0.0
2015-11-10 23:26:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=329, evicted=0, evictedPerRun=0.0
2015-11-10 23:31:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=359, evicted=0, evictedPerRun=0.0
2015-11-10 23:31:01,096 INFO org.apache.hadoop.hbase.mob.MobFileCache:
MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted
files: 0
2015-11-10 23:36:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=389, evicted=0, evictedPerRun=0.0
2015-11-10 23:41:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=419, evicted=0, evictedPerRun=0.0
2015-11-10 23:46:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=449, evicted=0, evictedPerRun=0.0
2015-11-10 23:47:34,668 INFO
org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
10.125.122.237 set balanceSwitch=false
2015-11-10 23:47:44,843 INFO
org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
10.125.122.237 set balanceSwitch=true
2015-11-10 23:51:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=479, evicted=0, evictedPerRun=0.0
2015-11-10 23:56:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=509, evicted=0, evictedPerRun=0.0
2015-11-11 00:01:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=539, evicted=0, evictedPerRun=0.0
2015-11-11 00:06:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=569, evicted=0, evictedPerRun=0.0
2015-11-11 00:11:01,089 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=599, evicted=0, evictedPerRun=0.0
Region server logs
tail -100
hbase-cmf-hbase-REGIONSERVER-hadoop4-private.wdc01.infra.ripple.com.log.out
2015-11-10 22:31:16,255 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=3460104,
freeSize=3369704184, maxSize=3373164288, heapSize=3460104,
minSize=3204506112, minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2015-11-10 22:31:16,255 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
major jitter 0.500000, min locality to compact 0.000000
2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor:
Updated row prod_control,,1434077692105.a4f70c6c01f31f6d3d65e0cf55200c80.
with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor:
Updated row
prod_agg_account_payments,20140325000000|rH48e8bf5kNWuFRBwgm5sCH133cd9zgV29,1447197624013.c2a9f77791512281e4491e0797e824e8.
with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
2015-11-10 22:31:16,259 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=3460104,
freeSize=3369704184, maxSize=3373164288, heapSize=3460104,
minSize=3204506112, minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2015-11-10 22:31:16,259 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=3460104,
freeSize=3369704184, maxSize=3373164288, heapSize=3460104,
minSize=3204506112, minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2015-11-10 22:31:16,259 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
major jitter 0.500000, min locality to compact 0.000000
2015-11-10 22:31:16,259 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
major jitter 0.500000, min locality to compact 0.000000
2015-11-10 22:31:16,262 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=3460104,
freeSize=3369704184, maxSize=3373164288, heapSize=3460104,
minSize=3204506112, minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2015-11-10 22:31:16,262 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
major jitter 0.500000, min locality to compact 0.000000
2015-11-10 22:31:16,265 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=3460104,
freeSize=3369704184, maxSize=3373164288, heapSize=3460104,
minSize=3204506112, minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2015-11-10 22:31:16,265 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
major jitter 0.500000, min locality to compact 0.000000
2015-11-10 22:31:16,265 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=3460104,
freeSize=3369704184, maxSize=3373164288, heapSize=3460104,
minSize=3204506112, minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2015-11-10 22:31:16,265 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
major jitter 0.500000, min locality to compact 0.000000
2015-11-10 22:31:16,267 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=3460104,
freeSize=3369704184, maxSize=3373164288, heapSize=3460104,
minSize=3204506112, minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2015-11-10 22:31:16,268 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
major jitter 0.500000, min locality to compact 0.000000
2015-11-10 22:31:16,272 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=3460104,
freeSize=3369704184, maxSize=3373164288, heapSize=3460104,
minSize=3204506112, minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2015-11-10 22:31:16,273 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
major jitter 0.500000, min locality to compact 0.000000
2015-11-10 22:31:16,274 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Onlined 1fbbd37bdff125be32d352c98dffd2b1; next sequenceid=180083
2015-11-10 22:31:16,274 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Onlined 7ce823a4e0f56bf8c854175409bb17b6; next sequenceid=21535
2015-11-10 22:31:16,275 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
for
prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1.
2015-11-10 22:31:16,276 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
for
prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6.
2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor:
Updated row
prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1.
with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor:
Updated row
prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6.
with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
2015-11-10 22:31:16,282 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=3460104,
freeSize=3369704184, maxSize=3373164288, heapSize=3460104,
minSize=3204506112, minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2015-11-10 22:31:16,282 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=3460104,
freeSize=3369704184, maxSize=3373164288, heapSize=3460104,
minSize=3204506112, minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2015-11-10 22:31:16,282 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Onlined 15701b8131ca505f6cce148d49cfbeee; next sequenceid=1029
2015-11-10 22:31:16,283 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
major jitter 0.500000, min locality to compact 0.000000
2015-11-10 22:31:16,283 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
major jitter 0.500000, min locality to compact 0.000000
2015-11-10 22:31:16,284 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
for
prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee.
2015-11-10 22:31:16,304 INFO org.apache.hadoop.hbase.MetaTableAccessor:
Updated row
prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee.
with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
2015-11-10 22:31:16,308 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=3460104,
freeSize=3369704184, maxSize=3373164288, heapSize=3460104,
minSize=3204506112, minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2015-11-10 22:31:16,308 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
major jitter 0.500000, min locality to compact 0.000000
2015-11-10 22:31:16,308 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=3460104,
freeSize=3369704184, maxSize=3373164288, heapSize=3460104,
minSize=3204506112, minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2015-11-10 22:31:16,309 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
major jitter 0.500000, min locality to compact 0.000000
2015-11-10 22:31:16,318 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Onlined 21b24baeaf26e12d02f79f243076850a; next sequenceid=114628
2015-11-10 22:31:16,318 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Onlined cc52e0ba23708df10d696d0e33d9d137; next sequenceid=161394
2015-11-10 22:31:16,320 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
for
prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a.
2015-11-10 22:31:16,320 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
for
prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD|2014111212,1447198690089.cc52e0ba23708df10d696d0e33d9d137.
2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor:
Updated row
prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD|2014111212,1447198690089.cc52e0ba23708df10d696d0e33d9d137.
with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor:
Updated row
prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a.
with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
2015-11-10 22:31:21,184 WARN org.apache.hadoop.metrics2.lib.Interns:
Metrics intern cache overflow at 2011 for
RegionServer={RegionServer=MetricsInfoImpl{name=RegionServer,
description=RegionServer}, RegionServer
record=MetricsInfoImpl{name=RegionServer, description=RegionServer record}}
2015-11-10 22:31:22,377 WARN
org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil:
hbase.regionserver.global.memstore.upperLimit is deprecated by
hbase.regionserver.global.memstore.size
2015-11-10 22:31:22,403 INFO org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 3 (all) file(s) in f of
prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041.
into d98861e6b5364d819d11c3f3e1c8fb56(size=180.4 M), total size for store
is 180.4 M. This selection was in queue for 0sec, and took 7sec to execute.
2015-11-10 22:31:22,405 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
compaction: Request =
regionName=prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041.,
storeName=f, fileCount=3, fileSize=180.4 M, priority=27,
time=741524299592168; duration=7sec
2015-11-10 22:31:22,413 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on f in region
prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
2015-11-10 22:31:22,413 INFO org.apache.hadoop.hbase.regionserver.HStore:
Starting compaction of 3 file(s) in f of
prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
into tmpdir=hdfs://
hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_transactions/72f5a6c7dba1e486eae6b75f4e9f0d5d/.tmp,
totalSize=82.2 M
2015-11-10 22:31:22,416 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=3460104,
freeSize=3369704184, maxSize=3373164288, heapSize=3460104,
minSize=3204506112, minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2015-11-10 22:31:25,301 INFO org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 3 (all) file(s) in f of
prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
into f7678de8dd8149428938a0aedfcbcaf9(size=82.2 M), total size for store is
82.2 M. This selection was in queue for 0sec, and took 2sec to execute.
2015-11-10 22:31:25,302 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
compaction: Request =
regionName=prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.,
storeName=f, fileCount=3, fileSize=82.2 M, priority=27,
time=741532147583369; duration=2sec
2015-11-10 22:31:25,305 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on f in region
prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
2015-11-10 22:31:25,305 INFO org.apache.hadoop.hbase.regionserver.HStore:
Starting compaction of 3 file(s) in f of
prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
into tmpdir=hdfs://
hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_offers/05284b692219df79852f3deaa2ce1927/.tmp,
totalSize=218.4 M
2015-11-10 22:31:25,310 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=3460104,
freeSize=3369704184, maxSize=3373164288, heapSize=3460104,
minSize=3204506112, minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2015-11-10 22:31:39,481 INFO org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 3 (all) file(s) in f of
prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
into 4aea6e4301f848ff9bf727f4e511978a(size=214.6 M), total size for store
is 214.6 M. This selection was in queue for 0sec, and took 14sec to execute.
2015-11-10 22:31:39,481 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
compaction: Request =
regionName=prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.,
storeName=f, fileCount=3, fileSize=218.4 M, priority=27,
time=741535039619615; duration=14sec
2015-11-10 22:31:39,484 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting compaction on d in region
prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
2015-11-10 22:31:39,484 INFO org.apache.hadoop.hbase.regionserver.HStore:
Starting compaction of 3 file(s) in d of
prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
into tmpdir=hdfs://
hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_balance_changes/40d39088cc3efe2d230a7d6f8d7b205b/.tmp,
totalSize=197.7 M
2015-11-10 22:31:39,490 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=0, currentSize=3460104,
freeSize=3369704184, maxSize=3373164288, heapSize=3460104,
minSize=3204506112, minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2015-11-10 22:31:52,393 INFO org.apache.hadoop.hbase.regionserver.HStore:
Completed compaction of 3 (all) file(s) in d of
prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
into 1e48a57cdb544e898e78c9e8276ec05b(size=197.7 M), total size for store
is 197.7 M. This selection was in queue for 0sec, and took 12sec to execute.
2015-11-10 22:31:52,393 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
compaction: Request =
regionName=prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.,
storeName=d, fileCount=3, fileSize=197.7 M, priority=27,
time=741549219351203; duration=12sec
2015-11-10 22:32:01,402 INFO
org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Atomically
moving hadoop3-private.wdc01.infra.ripple.com,60020,1447102836904's wals to
my queue
2015-11-10 22:32:01,405 WARN
org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer 1 didn't
exist, skipping the replay
2015-11-10 22:32:01,406 WARN
org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer
1-hadoop4-private.wdc01.infra.ripple.com,60020,1447102248964 didn't exist,
skipping the replay
2015-11-10 22:32:01,412 WARN
org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Got exception
in copyQueuesFromRSUsingMulti:
org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode =
Directory not empty
at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1152)
at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:1118)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:685)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1550)
at
org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.copyQueuesFromRSUsingMulti(ReplicationQueuesZKImpl.java:340)
at
org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.claimQueues(ReplicationQueuesZKImpl.java:172)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:570)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-11-10 22:36:01,223 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=3.78 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=4, accesses=63330, hits=6,
hitRatio=0.01%, , cachingAccesses=21, cachingHits=4,
cachingHitsRatio=19.05%, evictions=29, evicted=3,
evictedPerRun=0.1034482792019844
2015-11-10 22:41:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.15 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=7, accesses=63856, hits=8,
hitRatio=0.01%, , cachingAccesses=32, cachingHits=6,
cachingHitsRatio=18.75%, evictions=59, evicted=3,
evictedPerRun=0.050847455859184265
2015-11-10 22:46:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
cachingHitsRatio=21.43%, evictions=89, evicted=3,
evictedPerRun=0.033707864582538605
2015-11-10 22:51:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
cachingHitsRatio=21.43%, evictions=119, evicted=3,
evictedPerRun=0.02521008439362049
2015-11-10 22:56:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
cachingHitsRatio=21.43%, evictions=149, evicted=3,
evictedPerRun=0.020134227350354195
2015-11-10 23:01:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63872, hits=13,
hitRatio=0.02%, , cachingAccesses=48, cachingHits=11,
cachingHitsRatio=22.92%, evictions=179, evicted=3,
evictedPerRun=0.016759777441620827
2015-11-10 23:06:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63890, hits=21,
hitRatio=0.03%, , cachingAccesses=66, cachingHits=19,
cachingHitsRatio=28.79%, evictions=209, evicted=3,
evictedPerRun=0.014354066923260689
2015-11-10 23:11:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64416, hits=26,
hitRatio=0.04%, , cachingAccesses=77, cachingHits=24,
cachingHitsRatio=31.17%, evictions=239, evicted=3,
evictedPerRun=0.012552301399409771
2015-11-10 23:16:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
cachingHitsRatio=33.33%, evictions=269, evicted=3,
evictedPerRun=0.011152416467666626
2015-11-10 23:21:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
cachingHitsRatio=33.33%, evictions=299, evicted=3,
evictedPerRun=0.010033444501459599
2015-11-10 23:26:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
cachingHitsRatio=33.33%, evictions=329, evicted=3,
evictedPerRun=0.009118541143834591
2015-11-10 23:31:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64432, hits=34,
hitRatio=0.05%, , cachingAccesses=93, cachingHits=32,
cachingHitsRatio=34.41%, evictions=359, evicted=3,
evictedPerRun=0.008356546051800251
2015-11-10 23:31:01,228 INFO org.apache.hadoop.hbase.mob.MobFileCache:
MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted
files: 0
2015-11-10 23:31:06,790 INFO
org.apache.hadoop.hbase.regionserver.wal.FSHLog: Rolled WAL /hbase/WALs/
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/
hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
with entries=247, filesize=125.29 KB; new WAL /hbase/WALs/
hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/
hadoop4-private.wdc01.infra.ripple.com
%2C60020%2C1447216261115.null0.1447219866745
2015-11-10 23:31:06,792 INFO
org.apache.hadoop.hbase.regionserver.wal.FSHLog: Archiving hdfs://
hadoop2-private.wdc01.infra.ripple.com:8020/hbase/WALs/hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
to hdfs://
hadoop2-private.wdc01.infra.ripple.com:8020/hbase/oldWALs/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
2015-11-10 23:36:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64450, hits=42,
hitRatio=0.07%, , cachingAccesses=111, cachingHits=40,
cachingHitsRatio=36.04%, evictions=389, evicted=3,
evictedPerRun=0.007712082471698523
2015-11-10 23:41:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64976, hits=47,
hitRatio=0.07%, , cachingAccesses=122, cachingHits=45,
cachingHitsRatio=36.89%, evictions=419, evicted=3,
evictedPerRun=0.0071599045768380165
2015-11-10 23:46:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
cachingHitsRatio=37.88%, evictions=449, evicted=3,
evictedPerRun=0.00668151443824172
2015-11-10 23:51:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
cachingHitsRatio=37.88%, evictions=479, evicted=3,
evictedPerRun=0.00626304792240262
2015-11-10 23:56:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
cachingHitsRatio=37.88%, evictions=509, evicted=3,
evictedPerRun=0.005893909838050604
2015-11-11 00:01:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64992, hits=55,
hitRatio=0.08%, , cachingAccesses=138, cachingHits=53,
cachingHitsRatio=38.41%, evictions=539, evicted=3,
evictedPerRun=0.005565862637013197
2015-11-11 00:06:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65010, hits=63,
hitRatio=0.10%, , cachingAccesses=156, cachingHits=61,
cachingHitsRatio=39.10%, evictions=569, evicted=3,
evictedPerRun=0.005272407550364733
2015-11-11 00:11:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65536, hits=68,
hitRatio=0.10%, , cachingAccesses=167, cachingHits=66,
cachingHitsRatio=39.52%, evictions=599, evicted=3,
evictedPerRun=0.00500834733247757
2015-11-11 00:16:01,222 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65546, hits=73,
hitRatio=0.11%, , cachingAccesses=177, cachingHits=71,
cachingHitsRatio=40.11%, evictions=629, evicted=3,
evictedPerRun=0.004769475199282169
On Tue, Nov 10, 2015 at 8:15 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. The region exits are forcing the data to start transferring to other
> regions,
>
> Can you elaborate a bit more on the above ? Did you mean some regions were
> transferred to other region server(s) ?
>
> Can you determined the effective RegionSplitPolicy ?
>
> Snippets of server logs would help illustrate the situation.
>
> Thanks
>
> On Tue, Nov 10, 2015 at 8:09 PM, Abraham Tom <wo...@gmail.com> wrote:
>
>> I am on CDH 5.4.8 hbase 1.0
>>
>> I am migrating to a new cluster and the data is sourced from CDH 5.3 hbase
>> 0.98.6
>>
>> I took snapshots from the old cluster exported them to the new cluster -
>> no
>> errors
>>
>> cloned the table, flush, major_compact and balanced the regions
>>
>> the configs are practically identical to the old cluster
>>
>>
>>
>> hbck and fsck all check out with no inconsistencies found
>>
>>
>>
>> yet when we try to load data via thrift the regions start splitting at an
>> outrageous pace.
>>
>> Eventually the regions become offline starting a snowball affect. The
>> region exits are forcing the data to start transferring to other regions,
>> with that creates large GC activity and multiple regions in transition
>> before thrift can no longer execute any writes. This usually happens
>> after
>> a few records. How we put is that we perform rowkey scans before puts to
>> ensure where we last left off.
>>
>>
>>
>> Thrift eventually stops writing due to regions being offline and no longer
>> serving. The regions eventually stop splitting and all the GC eventually
>> complete. Once things settle back down, I am able to perform hbck and
>> fsck
>> and it again comes back with no inconsistencies
>>
>>
>>
>> I need ideas as to what is causing this
>>
>>
>> --
>> Abraham Tom
>> Email: work2much@gmail.com
>> Phone: 415-515-3621
>>
>
>
--
Abraham Tom
Email: work2much@gmail.com
Phone: 415-515-3621
Re: hbase regions behaving badly
Posted by Ted Yu <yu...@gmail.com>.
bq. The region exits are forcing the data to start transferring to other
regions,
Can you elaborate a bit more on the above ? Did you mean some regions were
transferred to other region server(s) ?
Can you determined the effective RegionSplitPolicy ?
Snippets of server logs would help illustrate the situation.
Thanks
On Tue, Nov 10, 2015 at 8:09 PM, Abraham Tom <wo...@gmail.com> wrote:
> I am on CDH 5.4.8 hbase 1.0
>
> I am migrating to a new cluster and the data is sourced from CDH 5.3 hbase
> 0.98.6
>
> I took snapshots from the old cluster exported them to the new cluster - no
> errors
>
> cloned the table, flush, major_compact and balanced the regions
>
> the configs are practically identical to the old cluster
>
>
>
> hbck and fsck all check out with no inconsistencies found
>
>
>
> yet when we try to load data via thrift the regions start splitting at an
> outrageous pace.
>
> Eventually the regions become offline starting a snowball affect. The
> region exits are forcing the data to start transferring to other regions,
> with that creates large GC activity and multiple regions in transition
> before thrift can no longer execute any writes. This usually happens after
> a few records. How we put is that we perform rowkey scans before puts to
> ensure where we last left off.
>
>
>
> Thrift eventually stops writing due to regions being offline and no longer
> serving. The regions eventually stop splitting and all the GC eventually
> complete. Once things settle back down, I am able to perform hbck and fsck
> and it again comes back with no inconsistencies
>
>
>
> I need ideas as to what is causing this
>
>
> --
> Abraham Tom
> Email: work2much@gmail.com
> Phone: 415-515-3621
>