You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by yun peng <pe...@gmail.com> on 2012/11/08 07:08:12 UTC
How to check if a major_compact is done?
Hi, All,
I want to measure the duration of a major compaction in HBase. Since the
function call majorCompact is asynchronous, I may need to manually check
when the major compaction is done. Does Hbase (as of version 0.92.4)
provide an interface to determine completion of major compaction? Thanks.
Yun
Re: How to check if a major_compact is done?
Posted by Jerry Lam <ch...@gmail.com>.
Hi Yun:
Please refer to HBase Metric:
http://hbase.apache.org/book/hbase_metrics.html
The hbase.regionserver.compactionQueueSize seems promising but I'm not
certain because I have never use it.
Best Regards,
Jerry
On Thu, Nov 8, 2012 at 6:43 AM, Jean-Marc Spaggiari <jean-marc@spaggiari.org
> wrote:
> Please someone correct me if I'm wrong, but I think there is some
> information exposed to JMX which give you the duration (and size) of
> the last compaction.
>
> JM
>
> 2012/11/8, PG <pe...@gmail.com>:
> > Hi, thanks for the comments. One thing is,,,, shouldn't web UI comes from
> > the hbase API, or can I issue function call to get the progress of
> > compaction?.
> > Hun
> >
> > On Nov 8, 2012, at 1:33 AM, ramkrishna vasudevan
> > <ra...@gmail.com> wrote:
> >
> >> There is no interface which says that the major compaction is completed.
> >> But you can see that major compaction is in progress from the web UI.
> >> Sorry if am wrong here.
> >>
> >> Regards
> >> Ram
> >>
> >> On Thu, Nov 8, 2012 at 11:38 AM, yun peng <pe...@gmail.com>
> wrote:
> >>
> >>> Hi, All,
> >>> I want to measure the duration of a major compaction in HBase. Since
> the
> >>> function call majorCompact is asynchronous, I may need to manually
> check
> >>> when the major compaction is done. Does Hbase (as of version 0.92.4)
> >>> provide an interface to determine completion of major compaction?
> >>> Thanks.
> >>> Yun
> >>>
> >
>
Re: How to check if a major_compact is done?
Posted by Jeremy Carroll <ph...@gmail.com>.
The JIRA ticket is as follows.
https://issues.apache.org/jira/browse/HBASE-6033 (Original)
https://issues.apache.org/jira/browse/HBASE-6124 (Backport to 0.92.1, etc..)
On Thu, Nov 8, 2012 at 11:19 AM, Jeremy Carroll <ph...@gmail.com> wrote:
> What I have done is using HBase 0.92.2+ (I believe commited in HBase 0.94)
> there is a compaction status on the table via the Web-UI. I do not know if
> it's exposed anywhere else. For my own internal compaction tools, I scrape
> that status from the web UI to determine if a table is in minor / major /
> none compaction.
>
>
> https://s3.amazonaws.com/uploads.hipchat.com/10509/150181/39jhoklr70xbmfj/Screen%20Shot%202012-11-08%20at%2011.19.32%20AM.png
>
>
> On Thu, Nov 8, 2012 at 10:31 AM, yun peng <pe...@gmail.com> wrote:
>
>> Yes, JMX exposes compaction time. My cluster has JMX enabled, and I can
>> view certain statistics from remote jconsole.
>>
>> However, I dont find any reference talking about how to write a Java
>> program to collect a specific statistics, like compaction time available
>> in
>> region server status. Does anyone here have experience in writing
>> low-level
>> code to collect JMX data, instead of using jconsole or other high-level
>> tools...
>>
>> PS: I have found some sample code
>> here<
>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics/package-summary.html
>> >,
>> yet the API is deprecated.
>> Regards,
>> Yun
>> On Thu, Nov 8, 2012 at 6:43 AM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org
>> > wrote:
>>
>> > Please someone correct me if I'm wrong, but I think there is some
>> > information exposed to JMX which give you the duration (and size) of
>> > the last compaction.
>> >
>> > JM
>> >
>> > 2012/11/8, PG <pe...@gmail.com>:
>> > > Hi, thanks for the comments. One thing is,,,, shouldn't web UI comes
>> from
>> > > the hbase API, or can I issue function call to get the progress of
>> > > compaction?.
>> > > Hun
>> > >
>> > > On Nov 8, 2012, at 1:33 AM, ramkrishna vasudevan
>> > > <ra...@gmail.com> wrote:
>> > >
>> > >> There is no interface which says that the major compaction is
>> completed.
>> > >> But you can see that major compaction is in progress from the web UI.
>> > >> Sorry if am wrong here.
>> > >>
>> > >> Regards
>> > >> Ram
>> > >>
>> > >> On Thu, Nov 8, 2012 at 11:38 AM, yun peng <pe...@gmail.com>
>> > wrote:
>> > >>
>> > >>> Hi, All,
>> > >>> I want to measure the duration of a major compaction in HBase. Since
>> > the
>> > >>> function call majorCompact is asynchronous, I may need to manually
>> > check
>> > >>> when the major compaction is done. Does Hbase (as of version 0.92.4)
>> > >>> provide an interface to determine completion of major compaction?
>> > >>> Thanks.
>> > >>> Yun
>> > >>>
>> > >
>> >
>>
>
>
Re: How to check if a major_compact is done?
Posted by Jeremy Carroll <ph...@gmail.com>.
What I have done is using HBase 0.92.2+ (I believe commited in HBase 0.94)
there is a compaction status on the table via the Web-UI. I do not know if
it's exposed anywhere else. For my own internal compaction tools, I scrape
that status from the web UI to determine if a table is in minor / major /
none compaction.
https://s3.amazonaws.com/uploads.hipchat.com/10509/150181/39jhoklr70xbmfj/Screen%20Shot%202012-11-08%20at%2011.19.32%20AM.png
On Thu, Nov 8, 2012 at 10:31 AM, yun peng <pe...@gmail.com> wrote:
> Yes, JMX exposes compaction time. My cluster has JMX enabled, and I can
> view certain statistics from remote jconsole.
>
> However, I dont find any reference talking about how to write a Java
> program to collect a specific statistics, like compaction time available in
> region server status. Does anyone here have experience in writing low-level
> code to collect JMX data, instead of using jconsole or other high-level
> tools...
>
> PS: I have found some sample code
> here<
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics/package-summary.html
> >,
> yet the API is deprecated.
> Regards,
> Yun
> On Thu, Nov 8, 2012 at 6:43 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org
> > wrote:
>
> > Please someone correct me if I'm wrong, but I think there is some
> > information exposed to JMX which give you the duration (and size) of
> > the last compaction.
> >
> > JM
> >
> > 2012/11/8, PG <pe...@gmail.com>:
> > > Hi, thanks for the comments. One thing is,,,, shouldn't web UI comes
> from
> > > the hbase API, or can I issue function call to get the progress of
> > > compaction?.
> > > Hun
> > >
> > > On Nov 8, 2012, at 1:33 AM, ramkrishna vasudevan
> > > <ra...@gmail.com> wrote:
> > >
> > >> There is no interface which says that the major compaction is
> completed.
> > >> But you can see that major compaction is in progress from the web UI.
> > >> Sorry if am wrong here.
> > >>
> > >> Regards
> > >> Ram
> > >>
> > >> On Thu, Nov 8, 2012 at 11:38 AM, yun peng <pe...@gmail.com>
> > wrote:
> > >>
> > >>> Hi, All,
> > >>> I want to measure the duration of a major compaction in HBase. Since
> > the
> > >>> function call majorCompact is asynchronous, I may need to manually
> > check
> > >>> when the major compaction is done. Does Hbase (as of version 0.92.4)
> > >>> provide an interface to determine completion of major compaction?
> > >>> Thanks.
> > >>> Yun
> > >>>
> > >
> >
>
Re: How to check if a major_compact is done?
Posted by yun peng <pe...@gmail.com>.
Hi, I have fixed the problem. I found the
MBeanServerConnection.getAttribute(ObjectName name, String attribute) is
the one useful here. Thanks to all.
Yun
On Thu, Nov 8, 2012 at 3:58 PM, yun peng <pe...@gmail.com> wrote:
> Hi, Mikael and Jeremy, thanks for your detailed answers. I have tried the
> MBean code as an attempt. I get certain information out from the printout,
> like follows,
> ...
> javax.management.MBeanAttributeInfo[description=NoDescription,
> name=compactionTimeNumOps, type=java.lang.Integer, read-only, descriptor={}]
> ...
>
> I have a minor question, (sorry about the extra questions, but I am sort
> of newbie to web service interfaces). The descriptor (or the content) of
> MBeanAttributeInfo here is empty, which does not match with the data I can
> see through jconsole, which show compactionTimeNumOps=4 (after issuing two
> major compaction requests). Should I pause the data/content from somewhere
> other than MBeanAttributeInfo.getDescriptor().
>
> Regards,
> Yun
>
>
>
> On Thu, Nov 8, 2012 at 2:36 PM, Mikael Sitruk <mi...@gmail.com>wrote:
>
>> Yun hi
>>
>> I did something like this in the past (in 0.92.0 version). That trigger
>> major compaction and monitor their completion. I did not go through the
>> JMX
>> (which is not an indicator of major compaction), but via
>> the org.apache.hadoop.hbase.HServerLoad.RegionLoad
>> class.
>> When an major compaction ends the number of store file will be the number
>> of store which can be checked via: regionLoad.getStorefiles() ==
>> task.getStores()
>>
>> I will try to find the code back and post it on github.
>>
>>
>> As for MBeans you can do somehting like:
>>
>> url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" + hostName + ":"
>> +
>> portNum + "/jmxrmi");
>> HashMap<String, String> env = new HashMap<String, String>();
>> JMXConnector jmxc = JMXConnectorFactory.connect(url, env);
>> MBeanServerConnection mbsc = jmxc.getMBeanServerConnection();
>> ObjectName rsStat = new ObjectName(
>> "hadoop:service=RegionServer,name=RegionServerStatistics"); //put the
>> mbean you need here.
>> MBeanInfo mbeanInfo = mbsc.getMBeanInfo(rsStat);
>>
>> for (MBeanAttributeInfo mbAttrInfo : mbeanInfo.getAttributes()) {
>> System.out.println(mbAttrInfo);
>> }
>>
>>
>> Regards,
>> Mikael.S
>>
>> On Thu, Nov 8, 2012 at 8:31 PM, yun peng <pe...@gmail.com> wrote:
>>
>> > Yes, JMX exposes compaction time. My cluster has JMX enabled, and I can
>> > view certain statistics from remote jconsole.
>> >
>> > However, I dont find any reference talking about how to write a Java
>> > program to collect a specific statistics, like compaction time
>> available in
>> > region server status. Does anyone here have experience in writing
>> low-level
>> > code to collect JMX data, instead of using jconsole or other high-level
>> > tools...
>> >
>> > PS: I have found some sample code
>> > here<
>> >
>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics/package-summary.html
>> > >,
>> > yet the API is deprecated.
>> > Regards,
>> > Yun
>> > On Thu, Nov 8, 2012 at 6:43 AM, Jean-Marc Spaggiari <
>> > jean-marc@spaggiari.org
>> > > wrote:
>> >
>> > > Please someone correct me if I'm wrong, but I think there is some
>> > > information exposed to JMX which give you the duration (and size) of
>> > > the last compaction.
>> > >
>> > > JM
>> > >
>> > > 2012/11/8, PG <pe...@gmail.com>:
>> > > > Hi, thanks for the comments. One thing is,,,, shouldn't web UI comes
>> > from
>> > > > the hbase API, or can I issue function call to get the progress of
>> > > > compaction?.
>> > > > Hun
>> > > >
>> > > > On Nov 8, 2012, at 1:33 AM, ramkrishna vasudevan
>> > > > <ra...@gmail.com> wrote:
>> > > >
>> > > >> There is no interface which says that the major compaction is
>> > completed.
>> > > >> But you can see that major compaction is in progress from the web
>> UI.
>> > > >> Sorry if am wrong here.
>> > > >>
>> > > >> Regards
>> > > >> Ram
>> > > >>
>> > > >> On Thu, Nov 8, 2012 at 11:38 AM, yun peng <pe...@gmail.com>
>> > > wrote:
>> > > >>
>> > > >>> Hi, All,
>> > > >>> I want to measure the duration of a major compaction in HBase.
>> Since
>> > > the
>> > > >>> function call majorCompact is asynchronous, I may need to manually
>> > > check
>> > > >>> when the major compaction is done. Does Hbase (as of version
>> 0.92.4)
>> > > >>> provide an interface to determine completion of major compaction?
>> > > >>> Thanks.
>> > > >>> Yun
>> > > >>>
>> > > >
>> > >
>> >
>>
>
>
Re: How to check if a major_compact is done?
Posted by yun peng <pe...@gmail.com>.
Hi, Mikael and Jeremy, thanks for your detailed answers. I have tried the
MBean code as an attempt. I get certain information out from the printout,
like follows,
...
javax.management.MBeanAttributeInfo[description=NoDescription,
name=compactionTimeNumOps, type=java.lang.Integer, read-only, descriptor={}]
...
I have a minor question, (sorry about the extra questions, but I am sort of
newbie to web service interfaces). The descriptor (or the content) of
MBeanAttributeInfo here is empty, which does not match with the data I can
see through jconsole, which show compactionTimeNumOps=4 (after issuing two
major compaction requests). Should I pause the data/content from somewhere
other than MBeanAttributeInfo.getDescriptor().
Regards,
Yun
On Thu, Nov 8, 2012 at 2:36 PM, Mikael Sitruk <mi...@gmail.com>wrote:
> Yun hi
>
> I did something like this in the past (in 0.92.0 version). That trigger
> major compaction and monitor their completion. I did not go through the JMX
> (which is not an indicator of major compaction), but via
> the org.apache.hadoop.hbase.HServerLoad.RegionLoad
> class.
> When an major compaction ends the number of store file will be the number
> of store which can be checked via: regionLoad.getStorefiles() ==
> task.getStores()
>
> I will try to find the code back and post it on github.
>
>
> As for MBeans you can do somehting like:
>
> url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" + hostName + ":" +
> portNum + "/jmxrmi");
> HashMap<String, String> env = new HashMap<String, String>();
> JMXConnector jmxc = JMXConnectorFactory.connect(url, env);
> MBeanServerConnection mbsc = jmxc.getMBeanServerConnection();
> ObjectName rsStat = new ObjectName(
> "hadoop:service=RegionServer,name=RegionServerStatistics"); //put the
> mbean you need here.
> MBeanInfo mbeanInfo = mbsc.getMBeanInfo(rsStat);
>
> for (MBeanAttributeInfo mbAttrInfo : mbeanInfo.getAttributes()) {
> System.out.println(mbAttrInfo);
> }
>
>
> Regards,
> Mikael.S
>
> On Thu, Nov 8, 2012 at 8:31 PM, yun peng <pe...@gmail.com> wrote:
>
> > Yes, JMX exposes compaction time. My cluster has JMX enabled, and I can
> > view certain statistics from remote jconsole.
> >
> > However, I dont find any reference talking about how to write a Java
> > program to collect a specific statistics, like compaction time available
> in
> > region server status. Does anyone here have experience in writing
> low-level
> > code to collect JMX data, instead of using jconsole or other high-level
> > tools...
> >
> > PS: I have found some sample code
> > here<
> >
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics/package-summary.html
> > >,
> > yet the API is deprecated.
> > Regards,
> > Yun
> > On Thu, Nov 8, 2012 at 6:43 AM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org
> > > wrote:
> >
> > > Please someone correct me if I'm wrong, but I think there is some
> > > information exposed to JMX which give you the duration (and size) of
> > > the last compaction.
> > >
> > > JM
> > >
> > > 2012/11/8, PG <pe...@gmail.com>:
> > > > Hi, thanks for the comments. One thing is,,,, shouldn't web UI comes
> > from
> > > > the hbase API, or can I issue function call to get the progress of
> > > > compaction?.
> > > > Hun
> > > >
> > > > On Nov 8, 2012, at 1:33 AM, ramkrishna vasudevan
> > > > <ra...@gmail.com> wrote:
> > > >
> > > >> There is no interface which says that the major compaction is
> > completed.
> > > >> But you can see that major compaction is in progress from the web
> UI.
> > > >> Sorry if am wrong here.
> > > >>
> > > >> Regards
> > > >> Ram
> > > >>
> > > >> On Thu, Nov 8, 2012 at 11:38 AM, yun peng <pe...@gmail.com>
> > > wrote:
> > > >>
> > > >>> Hi, All,
> > > >>> I want to measure the duration of a major compaction in HBase.
> Since
> > > the
> > > >>> function call majorCompact is asynchronous, I may need to manually
> > > check
> > > >>> when the major compaction is done. Does Hbase (as of version
> 0.92.4)
> > > >>> provide an interface to determine completion of major compaction?
> > > >>> Thanks.
> > > >>> Yun
> > > >>>
> > > >
> > >
> >
>
Re: How to check if a major_compact is done?
Posted by yun peng <pe...@gmail.com>.
Hi, as I recently find out, by spinning on getStorefiles ==
task.getStores() may not be a robust method to check if
a compaction is done. For instance, if the interval for checking is set too
large and there are compaction and flushes
intertwined together, then in the end, getStorefiles() may not be equal to
task.getStores() even when compaction is done.
Currently, I have to disable flush(), but am open for other solutions....
Regards,
Yun
On Thu, Nov 8, 2012 at 2:36 PM, Mikael Sitruk <mi...@gmail.com>wrote:
> Yun hi
>
> I did something like this in the past (in 0.92.0 version). That trigger
> major compaction and monitor their completion. I did not go through the JMX
> (which is not an indicator of major compaction), but via
> the org.apache.hadoop.hbase.HServerLoad.RegionLoad
> class.
> When an major compaction ends the number of store file will be the number
> of store which can be checked via: regionLoad.getStorefiles() ==
> task.getStores()
>
> I will try to find the code back and post it on github.
>
>
> As for MBeans you can do somehting like:
>
> url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" + hostName + ":" +
> portNum + "/jmxrmi");
> HashMap<String, String> env = new HashMap<String, String>();
> JMXConnector jmxc = JMXConnectorFactory.connect(url, env);
> MBeanServerConnection mbsc = jmxc.getMBeanServerConnection();
> ObjectName rsStat = new ObjectName(
> "hadoop:service=RegionServer,name=RegionServerStatistics"); //put the
> mbean you need here.
> MBeanInfo mbeanInfo = mbsc.getMBeanInfo(rsStat);
>
> for (MBeanAttributeInfo mbAttrInfo : mbeanInfo.getAttributes()) {
> System.out.println(mbAttrInfo);
> }
>
>
> Regards,
> Mikael.S
>
> On Thu, Nov 8, 2012 at 8:31 PM, yun peng <pe...@gmail.com> wrote:
>
> > Yes, JMX exposes compaction time. My cluster has JMX enabled, and I can
> > view certain statistics from remote jconsole.
> >
> > However, I dont find any reference talking about how to write a Java
> > program to collect a specific statistics, like compaction time available
> in
> > region server status. Does anyone here have experience in writing
> low-level
> > code to collect JMX data, instead of using jconsole or other high-level
> > tools...
> >
> > PS: I have found some sample code
> > here<
> >
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics/package-summary.html
> > >,
> > yet the API is deprecated.
> > Regards,
> > Yun
> > On Thu, Nov 8, 2012 at 6:43 AM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org
> > > wrote:
> >
> > > Please someone correct me if I'm wrong, but I think there is some
> > > information exposed to JMX which give you the duration (and size) of
> > > the last compaction.
> > >
> > > JM
> > >
> > > 2012/11/8, PG <pe...@gmail.com>:
> > > > Hi, thanks for the comments. One thing is,,,, shouldn't web UI comes
> > from
> > > > the hbase API, or can I issue function call to get the progress of
> > > > compaction?.
> > > > Hun
> > > >
> > > > On Nov 8, 2012, at 1:33 AM, ramkrishna vasudevan
> > > > <ra...@gmail.com> wrote:
> > > >
> > > >> There is no interface which says that the major compaction is
> > completed.
> > > >> But you can see that major compaction is in progress from the web
> UI.
> > > >> Sorry if am wrong here.
> > > >>
> > > >> Regards
> > > >> Ram
> > > >>
> > > >> On Thu, Nov 8, 2012 at 11:38 AM, yun peng <pe...@gmail.com>
> > > wrote:
> > > >>
> > > >>> Hi, All,
> > > >>> I want to measure the duration of a major compaction in HBase.
> Since
> > > the
> > > >>> function call majorCompact is asynchronous, I may need to manually
> > > check
> > > >>> when the major compaction is done. Does Hbase (as of version
> 0.92.4)
> > > >>> provide an interface to determine completion of major compaction?
> > > >>> Thanks.
> > > >>> Yun
> > > >>>
> > > >
> > >
> >
>
Re: How to check if a major_compact is done?
Posted by Jeremy Carroll <ph...@gmail.com>.
Sorry for spam. I have written metrics collection utilities via my GitHub
repo for JMX stats via the REST interface for each RegionServer.
https://github.com/phobos182/hadoop-hbase-tools
More specificially
https://github.com/phobos182/hadoop-hbase-tools/blob/master/hbase/regionserver_metrics.rb
With 0.94 changing the JMX stats to unify the web-ui stats, I will be
rewriting these to use JMX.
On Thu, Nov 8, 2012 at 10:31 AM, yun peng <pe...@gmail.com> wrote:
> Yes, JMX exposes compaction time. My cluster has JMX enabled, and I can
> view certain statistics from remote jconsole.
>
> However, I dont find any reference talking about how to write a Java
> program to collect a specific statistics, like compaction time available in
> region server status. Does anyone here have experience in writing low-level
> code to collect JMX data, instead of using jconsole or other high-level
> tools...
>
> PS: I have found some sample code
> here<
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics/package-summary.html
> >,
> yet the API is deprecated.
> Regards,
> Yun
> On Thu, Nov 8, 2012 at 6:43 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org
> > wrote:
>
> > Please someone correct me if I'm wrong, but I think there is some
> > information exposed to JMX which give you the duration (and size) of
> > the last compaction.
> >
> > JM
> >
> > 2012/11/8, PG <pe...@gmail.com>:
> > > Hi, thanks for the comments. One thing is,,,, shouldn't web UI comes
> from
> > > the hbase API, or can I issue function call to get the progress of
> > > compaction?.
> > > Hun
> > >
> > > On Nov 8, 2012, at 1:33 AM, ramkrishna vasudevan
> > > <ra...@gmail.com> wrote:
> > >
> > >> There is no interface which says that the major compaction is
> completed.
> > >> But you can see that major compaction is in progress from the web UI.
> > >> Sorry if am wrong here.
> > >>
> > >> Regards
> > >> Ram
> > >>
> > >> On Thu, Nov 8, 2012 at 11:38 AM, yun peng <pe...@gmail.com>
> > wrote:
> > >>
> > >>> Hi, All,
> > >>> I want to measure the duration of a major compaction in HBase. Since
> > the
> > >>> function call majorCompact is asynchronous, I may need to manually
> > check
> > >>> when the major compaction is done. Does Hbase (as of version 0.92.4)
> > >>> provide an interface to determine completion of major compaction?
> > >>> Thanks.
> > >>> Yun
> > >>>
> > >
> >
>
Re: How to check if a major_compact is done?
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
https://github.com/aravind/hbase_compact may have what you need (thanks
Jeremy)
Otis
--
Search Analytics - http://sematext.com/search-analytics/index.html
Performance Monitoring - http://sematext.com/spm/index.html
On Thu, Nov 8, 2012 at 1:31 PM, yun peng <pe...@gmail.com> wrote:
> Yes, JMX exposes compaction time. My cluster has JMX enabled, and I can
> view certain statistics from remote jconsole.
>
> However, I dont find any reference talking about how to write a Java
> program to collect a specific statistics, like compaction time available in
> region server status. Does anyone here have experience in writing low-level
> code to collect JMX data, instead of using jconsole or other high-level
> tools...
>
> PS: I have found some sample code
> here<
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics/package-summary.html
> >,
> yet the API is deprecated.
> Regards,
> Yun
> On Thu, Nov 8, 2012 at 6:43 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org
> > wrote:
>
> > Please someone correct me if I'm wrong, but I think there is some
> > information exposed to JMX which give you the duration (and size) of
> > the last compaction.
> >
> > JM
> >
> > 2012/11/8, PG <pe...@gmail.com>:
> > > Hi, thanks for the comments. One thing is,,,, shouldn't web UI comes
> from
> > > the hbase API, or can I issue function call to get the progress of
> > > compaction?.
> > > Hun
> > >
> > > On Nov 8, 2012, at 1:33 AM, ramkrishna vasudevan
> > > <ra...@gmail.com> wrote:
> > >
> > >> There is no interface which says that the major compaction is
> completed.
> > >> But you can see that major compaction is in progress from the web UI.
> > >> Sorry if am wrong here.
> > >>
> > >> Regards
> > >> Ram
> > >>
> > >> On Thu, Nov 8, 2012 at 11:38 AM, yun peng <pe...@gmail.com>
> > wrote:
> > >>
> > >>> Hi, All,
> > >>> I want to measure the duration of a major compaction in HBase. Since
> > the
> > >>> function call majorCompact is asynchronous, I may need to manually
> > check
> > >>> when the major compaction is done. Does Hbase (as of version 0.92.4)
> > >>> provide an interface to determine completion of major compaction?
> > >>> Thanks.
> > >>> Yun
> > >>>
> > >
> >
>
Re: How to check if a major_compact is done?
Posted by Mikael Sitruk <mi...@gmail.com>.
Yun hi
I did something like this in the past (in 0.92.0 version). That trigger
major compaction and monitor their completion. I did not go through the JMX
(which is not an indicator of major compaction), but via
the org.apache.hadoop.hbase.HServerLoad.RegionLoad
class.
When an major compaction ends the number of store file will be the number
of store which can be checked via: regionLoad.getStorefiles() ==
task.getStores()
I will try to find the code back and post it on github.
As for MBeans you can do somehting like:
url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" + hostName + ":" +
portNum + "/jmxrmi");
HashMap<String, String> env = new HashMap<String, String>();
JMXConnector jmxc = JMXConnectorFactory.connect(url, env);
MBeanServerConnection mbsc = jmxc.getMBeanServerConnection();
ObjectName rsStat = new ObjectName(
"hadoop:service=RegionServer,name=RegionServerStatistics"); //put the
mbean you need here.
MBeanInfo mbeanInfo = mbsc.getMBeanInfo(rsStat);
for (MBeanAttributeInfo mbAttrInfo : mbeanInfo.getAttributes()) {
System.out.println(mbAttrInfo);
}
Regards,
Mikael.S
On Thu, Nov 8, 2012 at 8:31 PM, yun peng <pe...@gmail.com> wrote:
> Yes, JMX exposes compaction time. My cluster has JMX enabled, and I can
> view certain statistics from remote jconsole.
>
> However, I dont find any reference talking about how to write a Java
> program to collect a specific statistics, like compaction time available in
> region server status. Does anyone here have experience in writing low-level
> code to collect JMX data, instead of using jconsole or other high-level
> tools...
>
> PS: I have found some sample code
> here<
> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics/package-summary.html
> >,
> yet the API is deprecated.
> Regards,
> Yun
> On Thu, Nov 8, 2012 at 6:43 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org
> > wrote:
>
> > Please someone correct me if I'm wrong, but I think there is some
> > information exposed to JMX which give you the duration (and size) of
> > the last compaction.
> >
> > JM
> >
> > 2012/11/8, PG <pe...@gmail.com>:
> > > Hi, thanks for the comments. One thing is,,,, shouldn't web UI comes
> from
> > > the hbase API, or can I issue function call to get the progress of
> > > compaction?.
> > > Hun
> > >
> > > On Nov 8, 2012, at 1:33 AM, ramkrishna vasudevan
> > > <ra...@gmail.com> wrote:
> > >
> > >> There is no interface which says that the major compaction is
> completed.
> > >> But you can see that major compaction is in progress from the web UI.
> > >> Sorry if am wrong here.
> > >>
> > >> Regards
> > >> Ram
> > >>
> > >> On Thu, Nov 8, 2012 at 11:38 AM, yun peng <pe...@gmail.com>
> > wrote:
> > >>
> > >>> Hi, All,
> > >>> I want to measure the duration of a major compaction in HBase. Since
> > the
> > >>> function call majorCompact is asynchronous, I may need to manually
> > check
> > >>> when the major compaction is done. Does Hbase (as of version 0.92.4)
> > >>> provide an interface to determine completion of major compaction?
> > >>> Thanks.
> > >>> Yun
> > >>>
> > >
> >
>
Re: How to check if a major_compact is done?
Posted by yun peng <pe...@gmail.com>.
Yes, JMX exposes compaction time. My cluster has JMX enabled, and I can
view certain statistics from remote jconsole.
However, I dont find any reference talking about how to write a Java
program to collect a specific statistics, like compaction time available in
region server status. Does anyone here have experience in writing low-level
code to collect JMX data, instead of using jconsole or other high-level
tools...
PS: I have found some sample code
here<http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics/package-summary.html>,
yet the API is deprecated.
Regards,
Yun
On Thu, Nov 8, 2012 at 6:43 AM, Jean-Marc Spaggiari <jean-marc@spaggiari.org
> wrote:
> Please someone correct me if I'm wrong, but I think there is some
> information exposed to JMX which give you the duration (and size) of
> the last compaction.
>
> JM
>
> 2012/11/8, PG <pe...@gmail.com>:
> > Hi, thanks for the comments. One thing is,,,, shouldn't web UI comes from
> > the hbase API, or can I issue function call to get the progress of
> > compaction?.
> > Hun
> >
> > On Nov 8, 2012, at 1:33 AM, ramkrishna vasudevan
> > <ra...@gmail.com> wrote:
> >
> >> There is no interface which says that the major compaction is completed.
> >> But you can see that major compaction is in progress from the web UI.
> >> Sorry if am wrong here.
> >>
> >> Regards
> >> Ram
> >>
> >> On Thu, Nov 8, 2012 at 11:38 AM, yun peng <pe...@gmail.com>
> wrote:
> >>
> >>> Hi, All,
> >>> I want to measure the duration of a major compaction in HBase. Since
> the
> >>> function call majorCompact is asynchronous, I may need to manually
> check
> >>> when the major compaction is done. Does Hbase (as of version 0.92.4)
> >>> provide an interface to determine completion of major compaction?
> >>> Thanks.
> >>> Yun
> >>>
> >
>
Re: How to check if a major_compact is done?
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Please someone correct me if I'm wrong, but I think there is some
information exposed to JMX which give you the duration (and size) of
the last compaction.
JM
2012/11/8, PG <pe...@gmail.com>:
> Hi, thanks for the comments. One thing is,,,, shouldn't web UI comes from
> the hbase API, or can I issue function call to get the progress of
> compaction?.
> Hun
>
> On Nov 8, 2012, at 1:33 AM, ramkrishna vasudevan
> <ra...@gmail.com> wrote:
>
>> There is no interface which says that the major compaction is completed.
>> But you can see that major compaction is in progress from the web UI.
>> Sorry if am wrong here.
>>
>> Regards
>> Ram
>>
>> On Thu, Nov 8, 2012 at 11:38 AM, yun peng <pe...@gmail.com> wrote:
>>
>>> Hi, All,
>>> I want to measure the duration of a major compaction in HBase. Since the
>>> function call majorCompact is asynchronous, I may need to manually check
>>> when the major compaction is done. Does Hbase (as of version 0.92.4)
>>> provide an interface to determine completion of major compaction?
>>> Thanks.
>>> Yun
>>>
>
Re: How to check if a major_compact is done?
Posted by PG <pe...@gmail.com>.
Hi, thanks for the comments. One thing is,,,, shouldn't web UI comes from the hbase API, or can I issue function call to get the progress of compaction?.
Hun
On Nov 8, 2012, at 1:33 AM, ramkrishna vasudevan <ra...@gmail.com> wrote:
> There is no interface which says that the major compaction is completed.
> But you can see that major compaction is in progress from the web UI.
> Sorry if am wrong here.
>
> Regards
> Ram
>
> On Thu, Nov 8, 2012 at 11:38 AM, yun peng <pe...@gmail.com> wrote:
>
>> Hi, All,
>> I want to measure the duration of a major compaction in HBase. Since the
>> function call majorCompact is asynchronous, I may need to manually check
>> when the major compaction is done. Does Hbase (as of version 0.92.4)
>> provide an interface to determine completion of major compaction? Thanks.
>> Yun
>>
Re: How to check if a major_compact is done?
Posted by ramkrishna vasudevan <ra...@gmail.com>.
There is no interface which says that the major compaction is completed.
But you can see that major compaction is in progress from the web UI.
Sorry if am wrong here.
Regards
Ram
On Thu, Nov 8, 2012 at 11:38 AM, yun peng <pe...@gmail.com> wrote:
> Hi, All,
> I want to measure the duration of a major compaction in HBase. Since the
> function call majorCompact is asynchronous, I may need to manually check
> when the major compaction is done. Does Hbase (as of version 0.92.4)
> provide an interface to determine completion of major compaction? Thanks.
> Yun
>
Re: How to check if a major_compact is done?
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
I *think* SPM for HBase shows this. URL in signature. I'm guessing
Cloudera Manager may also show this if you are using CDH. Otherwise, you
can just read from JMX. Handy no-deps command-line tool for that:
https://github.com/sematext/jmxc
Otis
--
Performance Monitoring - http://sematext.com/spm/index.html
On Thu, Nov 8, 2012 at 1:08 AM, yun peng <pe...@gmail.com> wrote:
> Hi, All,
> I want to measure the duration of a major compaction in HBase. Since the
> function call majorCompact is asynchronous, I may need to manually check
> when the major compaction is done. Does Hbase (as of version 0.92.4)
> provide an interface to determine completion of major compaction? Thanks.
> Yun
>