You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by kzurek <kz...@proximetry.pl> on 2013/01/30 16:55:42 UTC
HBase 0.92.1 - triggering major compaction
Hi,
I'm having following issues with triggering manually major compaction on
selected regions via HBaseAdmin:
1. When I'm triggering major compaction on first region, which does not
contains key, it's running normally - I see message in logs ([..]Large
Compaction requested ... Because: User-triggered major compaction;[...]) and
after some time it's executed (info in logs). Example region name:
test,,1359115210217.7315770e499a47e0598849d8702ed202.
2. When I'm triggering major compaction on some chosen region (this one
contains a key) I see no information as before, like it would not be
triggered, although I'm not getting any errors or so. Thus, I'm not sure if
compaction which will be executed is major or rather minor. Moreover, I've
noticed that on one region which was previously (2h before) triggered for
compaction, I've received only information about the end of compaction
(Store: Completed major compaction of 3 file(s) in data of [...]). Example
region name:
test,\x00|@\x07\x00\x00\x0A\x19,1359529282226.aa16fae842a3e3f38a6ee75dc6a2941a.
3. How can I verify for sure that major compaction was really triggered?
I've also read on forum that since 0.92 version MC is triggered
asynchronously (is it?).
4. Why there is no such information in logs saying that major compaction
was triggered or is rolling (logs on DEBUG at the moment)?
5. Where can I find some detailed information about major compaction
itself?
Some basic information:
HBase Version: 0.92.1, r1298924
Hadoop Version: 1.0.3, r1335192
Automatic major compaction: off
Regards,
Krzysiek Z.
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-0-92-1-triggering-major-compaction-tp4037660.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase 0.92.1 - triggering major compaction
Posted by kzurek <kz...@proximetry.pl>.
Thanks for your reply. Below I've put some answers. Although, I'm now facing
other issues regarding to major and minor compaction. I've disabled major
compaction and it is not triggered manually, but when I'm loading data to
selected region, I saw that major compaction queue is growing and it is
being triggered ('Large Compaction' in logs) quite often. Moreover, I've
noticed that my clients app gets timeout while putting data into the cluster
(happens when memory store flusher is trying to dump memory content to store
file, but it cannot due to too many store files). For me, it looks like
compaction process is not fast enough comparing to incoming rate of data or
... (maybe something else?) and is blocking update process. I dont know if I
should create separate thread for this or not.
Some logs:
2013-02-01 15:43:14,627 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Large Compaction
requested:
regionName=test,\x00\x00\x00\x00~i\x91\x00\x00\x00\x0D,1359115210217.3b710693d6314c2a987b07dd82451158.,
storeName=data, fileCount=3, fileSize=478.3m (249.8m, 113.7m, 114.7m),
priority=-3, time=1051078047102762; Because: regionserver60020.cacheFlusher;
compaction_queue=(48:0), split_queue=0
2013-02-01 15:43:14,627 DEBUG org.apache.hadoop.hbase.regionserver.Store:
3b710693d6314c2a987b07dd82451158 - tag: no store files to compact
2013-02-01 15:43:14,709 WARN org.apache.hadoop.ipc.HBaseServer:
(responseTooSlow):
{"processingtimems":61908,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@57a56081),
rpc version=1, client version=29,
methodsFingerPrint=54742778","client":"192.168.1.68:49419","starttimems":1359729732799,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
2013-02-01 15:43:14,710 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder, call multi(org.apache.hadoop.hbase.client.MultiAction@57a56081),
rpc version=1, client version=29, methodsFingerPrint=54742778 from
192.168.1.68:49419: output error
2013-02-01 15:43:14,710 WARN org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60020 caught: java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelIO(HBaseServer.java:1710)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1653)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:924)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:1003)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBaseServer.java:409)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1346)
2013-02-01 15:43:19,397 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Flush requested on
test,\x00\x00\x00\x00~i\x91\x00\x00\x00\x0D,1359115210217.3b710693d6314c2a987b07dd82451158.
2013-02-01 15:43:19,397 WARN
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Region
test,\x00\x00\x00\x00~i\x91\x00\x00\x00\x0D,1359115210217.3b710693d6314c2a987b07dd82451158.
has too many store files; delaying flush up to 90000ms
2013-02-01 15:43:19,397 DEBUG org.apache.hadoop.hbase.regionserver.Store:
3b710693d6314c2a987b07dd82451158 - data: no store files to compact
2013-02-01 15:43:19,397 DEBUG org.apache.hadoop.hbase.regionserver.Store:
3b710693d6314c2a987b07dd82451158 - tag: no store files to compact
2013-02-01 15:43:55,693 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Blocking updates for 'IPC Server handler 10 on 60020' on region
test,\x00\x00\x00\x00~i\x91\x00\x00\x00\x0D,1359115210217.3b710693d6314c2a987b07dd82451158.:
memstore size 1.5g is >= than blocking 1.5g size
2013-02-01 15:44:49,452 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Waited 90055ms on a
compaction to clean up 'too many store files'; waited long enough...
proceeding with flush of
test,\x00\x00\x00\x00~i\x91\x00\x00\x00\x0D,1359115210217.3b710693d6314c2a987b07dd82451158.
2013-02-01 15:44:49,452 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memstore flush for
test,\x00\x00\x00\x00~i\x91\x00\x00\x00\x0D,1359115210217.3b710693d6314c2a987b07dd82451158.,
current region memstore size 1.5g
2013-02-01 15:44:49,453 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting
test,\x00\x00\x00\x00~i\x91\x00\x00\x00\x0D,1359115210217.3b710693d6314c2a987b07dd82451158.,
commencing wait for mvcc, flushsize=1624094576
2013-02-01 15:44:50,014 DEBUG
org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with
CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false]
[cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false]
[cacheEvictOnClose=false] [cacheCompressed=false]
2013-02-01 15:44:55,082 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Renaming flushed file at
hdfs://SOME-PC:9000/hbase/test/3b710693d6314c2a987b07dd82451158/.tmp/7ddee02376664ac387eb3c786c541ed0
to
hdfs://SOME-PC:9000/hbase/test/3b710693d6314c2a987b07dd82451158/data/7ddee02376664ac387eb3c786c541ed0
2013-02-01 15:44:55,086 INFO org.apache.hadoop.hbase.regionserver.Store:
Added
hdfs://SOME-PC:9000/hbase/test/3b710693d6314c2a987b07dd82451158/data/7ddee02376664ac387eb3c786c541ed0,
entries=7157100, sequenceid=58642, memsize=1.5g, filesize=113.8m
2013-02-01 15:44:55,087 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Unblocking updates for region
test,\x00\x00\x00\x00~i\x91\x00\x00\x00\x0D,1359115210217.3b710693d6314c2a987b07dd82451158.
'IPC Server handler 10 on 60020'
2013-02-01 15:44:55,087 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Finished memstore flush of ~1.5g/1624094576, currentsize=0.0/0 for region
test,\x00\x00\x00\x00~i\x91\x00\x00\x00\x0D,1359115210217.3b710693d6314c2a987b07dd82451158.
in 5635ms, sequenceid=58642, compaction requested=false
Jean-Daniel Cryans wrote
> On Wed, Jan 30, 2013 at 7:55 AM, kzurek <
> kzurek@
> > wrote:
>> Hi,
>>
>> I'm having following issues with triggering manually major compaction on
>> selected regions via HBaseAdmin:
>> 1. When I'm triggering major compaction on first region, which does not
>> contains key, it's running normally - I see message in logs ([..]Large
>> Compaction requested ... Because: User-triggered major compaction;[...])
>> and
>> after some time it's executed (info in logs). Example region name:
>> test,,1359115210217.7315770e499a47e0598849d8702ed202.
>
>> What do you mean by "does not contain key"? Are you saying that they
>> are simply empty or or are you starting major compactions using a row
>> key?
>
> I mean that I'm using first region to trigger major compaction - this one
> contains, let say, no key string in region name.
>
>> 2. When I'm triggering major compaction on some chosen region (this one
>> contains a key) I see no information as before, like it would not be
>> triggered, although I'm not getting any errors or so. Thus, I'm not sure
>> if
>> compaction which will be executed is major or rather minor. Moreover,
>> I've
>> noticed that on one region which was previously (2h before) triggered for
>> compaction, I've received only information about the end of compaction
>> (Store: Completed major compaction of 3 file(s) in data of [...]).
>> Example
>> region name:
>> test,\x00|@\x07\x00\x00\x0A\x19,1359529282226.aa16fae842a3e3f38a6ee75dc6a2941a.
>
>> Same question regarding "this one contains a key". But if you are
>> indeed in DEBUG-level then you should see the compaction being
>> requested. Can you post a log snippet in a pastebin.com (or similar)?
>
> The same as above, I'm using some other region with specified key row (one
> that region starts with) in its name.
>
>> 3. How can I verify for sure that major compaction was really triggered?
>> I've also read on forum that since 0.92 version MC is triggered
>> asynchronously (is it?).
>
>> IIRC it always was async, the client didn't wait on the compaction to
>> end before returning. The sure way to verify for sure if by looking at
>> the log, unfortunately.
>
> I thought so, thus during couple of last days I spent some time reading
> logs and its a little bit more clear right now.
>
>> 4. Why there is no such information in logs saying that major compaction
>> was triggered or is rolling (logs on DEBUG at the moment)?
>
> I guess there's something else we're missing but it's not like I can
> log into your cluster and verify it :) So log snippets are always
> useful.
>
>> 5. Where can I find some detailed information about major compaction
>> itself?
>
>> The best documentation is always the code itself, the book contains
>> some information regarding compactions in general but I don't think it
>> goes into the details.
> True, so I've looked at sources and right know is a little bit more clear
> than before. Moreover, I've recently discovered some explanation of
> compaction process (written by its author) on the forum.
>
>
> J-D
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-0-92-1-triggering-major-compaction-tp4037660p4037885.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase 0.92.1 - triggering major compaction
Posted by Jean-Daniel Cryans <jd...@apache.org>.
On Wed, Jan 30, 2013 at 7:55 AM, kzurek <kz...@proximetry.pl> wrote:
> Hi,
>
> I'm having following issues with triggering manually major compaction on
> selected regions via HBaseAdmin:
> 1. When I'm triggering major compaction on first region, which does not
> contains key, it's running normally - I see message in logs ([..]Large
> Compaction requested ... Because: User-triggered major compaction;[...]) and
> after some time it's executed (info in logs). Example region name:
> test,,1359115210217.7315770e499a47e0598849d8702ed202.
What do you mean by "does not contain key"? Are you saying that they
are simply empty or or are you starting major compactions using a row
key?
> 2. When I'm triggering major compaction on some chosen region (this one
> contains a key) I see no information as before, like it would not be
> triggered, although I'm not getting any errors or so. Thus, I'm not sure if
> compaction which will be executed is major or rather minor. Moreover, I've
> noticed that on one region which was previously (2h before) triggered for
> compaction, I've received only information about the end of compaction
> (Store: Completed major compaction of 3 file(s) in data of [...]). Example
> region name:
> test,\x00|@\x07\x00\x00\x0A\x19,1359529282226.aa16fae842a3e3f38a6ee75dc6a2941a.
Same question regarding "this one contains a key". But if you are
indeed in DEBUG-level then you should see the compaction being
requested. Can you post a log snippet in a pastebin.com (or similar)?
> 3. How can I verify for sure that major compaction was really triggered?
> I've also read on forum that since 0.92 version MC is triggered
> asynchronously (is it?).
IIRC it always was async, the client didn't wait on the compaction to
end before returning. The sure way to verify for sure if by looking at
the log, unfortunately.
> 4. Why there is no such information in logs saying that major compaction
> was triggered or is rolling (logs on DEBUG at the moment)?
I guess there's something else we're missing but it's not like I can
log into your cluster and verify it :) So log snippets are always
useful.
> 5. Where can I find some detailed information about major compaction
> itself?
The best documentation is always the code itself, the book contains
some information regarding compactions in general but I don't think it
goes into the details.
J-D