You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Geovanie Marquez <ge...@gmail.com> on 2014/08/11 18:49:12 UTC
Non-sensical? Compaction Metrics Hbase UI
What is the meaning of the progress value below?
I see the information broken down below in the following HBase UI.
http://region-server-ip:60030/rs-status?filter=general#regionCompactStats
Region Name Num.
<tablename>,`\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1398130030457.2dddb2b81d70ee4b4aaf39ca830beb41.
Compacting KVsNum.
13617
Compacted KVsCompaction
9221037
Progress
67717.09%
Another example:
Compacting KVsNum. Compacted KVsCompaction Progress
357792875 1624293960 453.98%
The progress seems non-sensical. How should I interpret it? I notice
something more than 100% most of the time.
Re: Non-sensical? Compaction Metrics Hbase UI
Posted by Geovanie Marquez <ge...@gmail.com>.
thanks for your response. Here I provide an example since I was looking
into it. Like you mentioned it can be done but it takes some effort.
I see this in the regionserver logs:
4:21:32.107 PM INFO org.apache.hadoop.hbase.regionserver.HRegion
Starting compaction on d in region
<table>,\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1407770330344.061ddbc4f79c44d6660d0aba6ded1323.
4:21:32.107 PM INFO org.apache.hadoop.hbase.regionserver.HStore
Starting compaction of 10 file(s) in d of
<table>,\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1407770330344.061ddbc4f79c44d6660d0aba6ded1323.
into tmpdir=hdfs://hmaster.in.wellcentive.com:8020/hbase/data/default/<table>/061ddbc4f79c44d6660d0aba6ded1323/.tmp,
totalSize=2.0 G
4:25:35.755 PM INFO org.apache.hadoop.hbase.regionserver.HStore
Completed compaction of 10 file(s) in d of
<table>,\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1407770330344.061ddbc4f79c44d6660d0aba6ded1323.
into e7471cf82de4417491e631f7b7f0d67a(size=2.0 G), total size for store is
51.4 G. This selection was in queue for 0sec, and took 4mins, 3sec to
execute.
4:25:35.755 PM INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread
Completed compaction: Request =
regionName=<table>,\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1407770330344.061ddbc4f79c44d6660d0aba6ded1323.,
storeName=d, fileCount=10, fileSize=2.0 G, priority=42,
time=4989389638797549; duration=4mins, 3sec
and I checked the HBase .tmp dir and after the logs show completed then I
can correlate the results. I can do this before the compaction completes by
process of elimination since the other file is much larger than 2GB, but it
would not be obvious if there were two files both smaller than 2GB.
[hdfs@hmaster gmarquez]$ hdfs dfs -ls
/hbase/data/default/<table>/061ddbc4f79c44d6660d0aba6ded1323/.tmp
Found 2 items
-rwxr-xr-x 3 hbase hbase 24561844224 2014-08-11 15:30
/hbase/data/default/<table>/061ddbc4f79c44d6660d0aba6ded1323/.tmp/22d1b6f857e4458d8f6c875d4f48a575
-rwxr-xr-x 3 hbase hbase 1744830464 2014-08-11 16:21
/hbase/data/default/<table>/061ddbc4f79c44d6660d0aba6ded1323/.tmp/e7471cf82de4417491e631f7b7f0d67a
process of elimination leads me to that second file name
'e7471cf82de4417491e631f7b7f0d67a' because the first one is already much
bigger than 2GB, but this is a bit tedious as you can tell.
On Mon, Aug 11, 2014 at 4:31 PM, Geovanie Marquez <
geovanie.marquez@gmail.com> wrote:
> nice, just took a look at this path in my system. I see the growth but
> that would only apply to major_compactions where all files are picked up
> and compacted because minor compactions won't be trackable this way without
> knowing which files were selected by the compaction algorithm.
>
> is there a method for getting progress on these?
>
>
>
> On Mon, Aug 11, 2014 at 3:14 PM, Bryan Beaudreault <
> bbeaudreault@hubspot.com> wrote:
>
>> I've also seen that on 0.94.x. The compaction progress is almost never
>> accurate.
>>
>> I've taken to doing:
>>
>> hdfs -du /hbase/<table>/<region>/.tmp
>>
>> to track the new hfile size, expecting it to be approximately the same
>> size
>> as total hfiles pre-compaction.
>>
>>
>> On Mon, Aug 11, 2014 at 3:11 PM, Geovanie Marquez <
>> geovanie.marquez@gmail.com> wrote:
>>
>> > *Attribute Name* *Value*
>> > HBase Version 0.98.1-cdh5.1.0, rUnknown
>> > HBase Compiled Sat Jul 12 08:20:49 PDT 2014, jenkins
>> > Hadoop Version 2.3.0-cdh5.1.0, r8e266e052e423af592871e2dfe09d54c03f6a0e8
>> > Hadoop Compiled 2014-07-12T13:49Z, jenkins
>> >
>> >
>> > On Mon, Aug 11, 2014 at 12:56 PM, Ted Yu <yu...@gmail.com> wrote:
>> >
>> > > Which release of hbase are you using ?
>> > >
>> > > Thanks
>> > >
>> > >
>> > > On Mon, Aug 11, 2014 at 9:49 AM, Geovanie Marquez <
>> > > geovanie.marquez@gmail.com> wrote:
>> > >
>> > > > What is the meaning of the progress value below?
>> > > >
>> > > >
>> > > >
>> > > > I see the information broken down below in the following HBase UI.
>> > > >
>> > > >
>> > >
>> >
>> http://region-server-ip:60030/rs-status?filter=general#regionCompactStats
>> > > >
>> > > >
>> > > >
>> > > > Region Name Num.
>> > > >
>> > > >
>> > > >
>> > >
>> >
>> <tablename>,`\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1398130030457.2dddb2b81d70ee4b4aaf39ca830beb41.
>> > > >
>> > > >
>> > > >
>> > > > Compacting KVsNum.
>> > > >
>> > > > 13617
>> > > >
>> > > >
>> > > >
>> > > > Compacted KVsCompaction
>> > > >
>> > > > 9221037
>> > > >
>> > > >
>> > > >
>> > > > Progress
>> > > >
>> > > > 67717.09%
>> > > >
>> > > >
>> > > >
>> > > > Another example:
>> > > >
>> > > > Compacting KVsNum. Compacted KVsCompaction Progress
>> > > >
>> > > > 357792875 1624293960 453.98%
>> > > >
>> > > >
>> > > > The progress seems non-sensical. How should I interpret it? I
>> notice
>> > > > something more than 100% most of the time.
>> > > >
>> > >
>> >
>>
>
>
Re: Non-sensical? Compaction Metrics Hbase UI
Posted by Bryan Beaudreault <bb...@hubspot.com>.
Unfortunately I don't know an easy way. But I think you could figure this
out by combining logs and the above. If you look at the logs on the server
compacting, it should show the number of files being compacted, the total
size, and the output file name. Then watch for the .tmp file growing to
approximately the totalSize listed in the logs.
Maybe someone else knows of a fix to the actual problem or an easier way to
achieve this.
On Mon, Aug 11, 2014 at 4:31 PM, Geovanie Marquez <
geovanie.marquez@gmail.com> wrote:
> nice, just took a look at this path in my system. I see the growth but that
> would only apply to major_compactions where all files are picked up and
> compacted because minor compactions won't be trackable this way without
> knowing which files were selected by the compaction algorithm.
>
> is there a method for getting progress on these?
>
>
>
> On Mon, Aug 11, 2014 at 3:14 PM, Bryan Beaudreault <
> bbeaudreault@hubspot.com
> > wrote:
>
> > I've also seen that on 0.94.x. The compaction progress is almost never
> > accurate.
> >
> > I've taken to doing:
> >
> > hdfs -du /hbase/<table>/<region>/.tmp
> >
> > to track the new hfile size, expecting it to be approximately the same
> size
> > as total hfiles pre-compaction.
> >
> >
> > On Mon, Aug 11, 2014 at 3:11 PM, Geovanie Marquez <
> > geovanie.marquez@gmail.com> wrote:
> >
> > > *Attribute Name* *Value*
> > > HBase Version 0.98.1-cdh5.1.0, rUnknown
> > > HBase Compiled Sat Jul 12 08:20:49 PDT 2014, jenkins
> > > Hadoop Version 2.3.0-cdh5.1.0,
> r8e266e052e423af592871e2dfe09d54c03f6a0e8
> > > Hadoop Compiled 2014-07-12T13:49Z, jenkins
> > >
> > >
> > > On Mon, Aug 11, 2014 at 12:56 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > Which release of hbase are you using ?
> > > >
> > > > Thanks
> > > >
> > > >
> > > > On Mon, Aug 11, 2014 at 9:49 AM, Geovanie Marquez <
> > > > geovanie.marquez@gmail.com> wrote:
> > > >
> > > > > What is the meaning of the progress value below?
> > > > >
> > > > >
> > > > >
> > > > > I see the information broken down below in the following HBase UI.
> > > > >
> > > > >
> > > >
> > >
> >
> http://region-server-ip:60030/rs-status?filter=general#regionCompactStats
> > > > >
> > > > >
> > > > >
> > > > > Region Name Num.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> <tablename>,`\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1398130030457.2dddb2b81d70ee4b4aaf39ca830beb41.
> > > > >
> > > > >
> > > > >
> > > > > Compacting KVsNum.
> > > > >
> > > > > 13617
> > > > >
> > > > >
> > > > >
> > > > > Compacted KVsCompaction
> > > > >
> > > > > 9221037
> > > > >
> > > > >
> > > > >
> > > > > Progress
> > > > >
> > > > > 67717.09%
> > > > >
> > > > >
> > > > >
> > > > > Another example:
> > > > >
> > > > > Compacting KVsNum. Compacted KVsCompaction Progress
> > > > >
> > > > > 357792875 1624293960 453.98%
> > > > >
> > > > >
> > > > > The progress seems non-sensical. How should I interpret it? I
> notice
> > > > > something more than 100% most of the time.
> > > > >
> > > >
> > >
> >
>
Re: Non-sensical? Compaction Metrics Hbase UI
Posted by Geovanie Marquez <ge...@gmail.com>.
nice, just took a look at this path in my system. I see the growth but that
would only apply to major_compactions where all files are picked up and
compacted because minor compactions won't be trackable this way without
knowing which files were selected by the compaction algorithm.
is there a method for getting progress on these?
On Mon, Aug 11, 2014 at 3:14 PM, Bryan Beaudreault <bbeaudreault@hubspot.com
> wrote:
> I've also seen that on 0.94.x. The compaction progress is almost never
> accurate.
>
> I've taken to doing:
>
> hdfs -du /hbase/<table>/<region>/.tmp
>
> to track the new hfile size, expecting it to be approximately the same size
> as total hfiles pre-compaction.
>
>
> On Mon, Aug 11, 2014 at 3:11 PM, Geovanie Marquez <
> geovanie.marquez@gmail.com> wrote:
>
> > *Attribute Name* *Value*
> > HBase Version 0.98.1-cdh5.1.0, rUnknown
> > HBase Compiled Sat Jul 12 08:20:49 PDT 2014, jenkins
> > Hadoop Version 2.3.0-cdh5.1.0, r8e266e052e423af592871e2dfe09d54c03f6a0e8
> > Hadoop Compiled 2014-07-12T13:49Z, jenkins
> >
> >
> > On Mon, Aug 11, 2014 at 12:56 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Which release of hbase are you using ?
> > >
> > > Thanks
> > >
> > >
> > > On Mon, Aug 11, 2014 at 9:49 AM, Geovanie Marquez <
> > > geovanie.marquez@gmail.com> wrote:
> > >
> > > > What is the meaning of the progress value below?
> > > >
> > > >
> > > >
> > > > I see the information broken down below in the following HBase UI.
> > > >
> > > >
> > >
> >
> http://region-server-ip:60030/rs-status?filter=general#regionCompactStats
> > > >
> > > >
> > > >
> > > > Region Name Num.
> > > >
> > > >
> > > >
> > >
> >
> <tablename>,`\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1398130030457.2dddb2b81d70ee4b4aaf39ca830beb41.
> > > >
> > > >
> > > >
> > > > Compacting KVsNum.
> > > >
> > > > 13617
> > > >
> > > >
> > > >
> > > > Compacted KVsCompaction
> > > >
> > > > 9221037
> > > >
> > > >
> > > >
> > > > Progress
> > > >
> > > > 67717.09%
> > > >
> > > >
> > > >
> > > > Another example:
> > > >
> > > > Compacting KVsNum. Compacted KVsCompaction Progress
> > > >
> > > > 357792875 1624293960 453.98%
> > > >
> > > >
> > > > The progress seems non-sensical. How should I interpret it? I notice
> > > > something more than 100% most of the time.
> > > >
> > >
> >
>
Re: Non-sensical? Compaction Metrics Hbase UI
Posted by Bryan Beaudreault <bb...@hubspot.com>.
I've also seen that on 0.94.x. The compaction progress is almost never
accurate.
I've taken to doing:
hdfs -du /hbase/<table>/<region>/.tmp
to track the new hfile size, expecting it to be approximately the same size
as total hfiles pre-compaction.
On Mon, Aug 11, 2014 at 3:11 PM, Geovanie Marquez <
geovanie.marquez@gmail.com> wrote:
> *Attribute Name* *Value*
> HBase Version 0.98.1-cdh5.1.0, rUnknown
> HBase Compiled Sat Jul 12 08:20:49 PDT 2014, jenkins
> Hadoop Version 2.3.0-cdh5.1.0, r8e266e052e423af592871e2dfe09d54c03f6a0e8
> Hadoop Compiled 2014-07-12T13:49Z, jenkins
>
>
> On Mon, Aug 11, 2014 at 12:56 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Which release of hbase are you using ?
> >
> > Thanks
> >
> >
> > On Mon, Aug 11, 2014 at 9:49 AM, Geovanie Marquez <
> > geovanie.marquez@gmail.com> wrote:
> >
> > > What is the meaning of the progress value below?
> > >
> > >
> > >
> > > I see the information broken down below in the following HBase UI.
> > >
> > >
> >
> http://region-server-ip:60030/rs-status?filter=general#regionCompactStats
> > >
> > >
> > >
> > > Region Name Num.
> > >
> > >
> > >
> >
> <tablename>,`\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1398130030457.2dddb2b81d70ee4b4aaf39ca830beb41.
> > >
> > >
> > >
> > > Compacting KVsNum.
> > >
> > > 13617
> > >
> > >
> > >
> > > Compacted KVsCompaction
> > >
> > > 9221037
> > >
> > >
> > >
> > > Progress
> > >
> > > 67717.09%
> > >
> > >
> > >
> > > Another example:
> > >
> > > Compacting KVsNum. Compacted KVsCompaction Progress
> > >
> > > 357792875 1624293960 453.98%
> > >
> > >
> > > The progress seems non-sensical. How should I interpret it? I notice
> > > something more than 100% most of the time.
> > >
> >
>
Re: Non-sensical? Compaction Metrics Hbase UI
Posted by Geovanie Marquez <ge...@gmail.com>.
*Attribute Name* *Value*
HBase Version 0.98.1-cdh5.1.0, rUnknown
HBase Compiled Sat Jul 12 08:20:49 PDT 2014, jenkins
Hadoop Version 2.3.0-cdh5.1.0, r8e266e052e423af592871e2dfe09d54c03f6a0e8
Hadoop Compiled 2014-07-12T13:49Z, jenkins
On Mon, Aug 11, 2014 at 12:56 PM, Ted Yu <yu...@gmail.com> wrote:
> Which release of hbase are you using ?
>
> Thanks
>
>
> On Mon, Aug 11, 2014 at 9:49 AM, Geovanie Marquez <
> geovanie.marquez@gmail.com> wrote:
>
> > What is the meaning of the progress value below?
> >
> >
> >
> > I see the information broken down below in the following HBase UI.
> >
> >
> http://region-server-ip:60030/rs-status?filter=general#regionCompactStats
> >
> >
> >
> > Region Name Num.
> >
> >
> >
> <tablename>,`\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1398130030457.2dddb2b81d70ee4b4aaf39ca830beb41.
> >
> >
> >
> > Compacting KVsNum.
> >
> > 13617
> >
> >
> >
> > Compacted KVsCompaction
> >
> > 9221037
> >
> >
> >
> > Progress
> >
> > 67717.09%
> >
> >
> >
> > Another example:
> >
> > Compacting KVsNum. Compacted KVsCompaction Progress
> >
> > 357792875 1624293960 453.98%
> >
> >
> > The progress seems non-sensical. How should I interpret it? I notice
> > something more than 100% most of the time.
> >
>
Re: Non-sensical? Compaction Metrics Hbase UI
Posted by Ted Yu <yu...@gmail.com>.
Which release of hbase are you using ?
Thanks
On Mon, Aug 11, 2014 at 9:49 AM, Geovanie Marquez <
geovanie.marquez@gmail.com> wrote:
> What is the meaning of the progress value below?
>
>
>
> I see the information broken down below in the following HBase UI.
>
> http://region-server-ip:60030/rs-status?filter=general#regionCompactStats
>
>
>
> Region Name Num.
>
>
> <tablename>,`\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1398130030457.2dddb2b81d70ee4b4aaf39ca830beb41.
>
>
>
> Compacting KVsNum.
>
> 13617
>
>
>
> Compacted KVsCompaction
>
> 9221037
>
>
>
> Progress
>
> 67717.09%
>
>
>
> Another example:
>
> Compacting KVsNum. Compacted KVsCompaction Progress
>
> 357792875 1624293960 453.98%
>
>
> The progress seems non-sensical. How should I interpret it? I notice
> something more than 100% most of the time.
>