You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by PRANEESH KUMAR <pr...@gmail.com> on 2015/06/10 06:55:45 UTC
How to make the client fast fail
Hi,
I have got the Connection object with default configuration, if the
zookeeper or HMaster or Region server is down, the client didn't fast fail
and it took almost 20 mins to thrown an error.
What is the best configuration to make the client fast fail.
Also what is significance of changing the following parameters.
hbase.client.retries.number
zookeeper.recovery.retry
zookeeper.session.timeout
zookeeper.recovery.retry.intervalmill
hbase.rpc.timeout
Regards,
Praneesh
Re: How to make the client fast fail
Posted by Michael Segel <mi...@hotmail.com>.
Be careful for what you wish.
You want to fail fast, ok, but when you shorten the HBase timers, you can run in to other problems.
The simplest solution is to use a timer / timeout thread in your application.
You want to do it this way because you are asking for an application specific solution while HBase is a shared resource.
Failing fast and failing often is no way to run an HBase/Hadoop cluster. ;-)
> On Jun 14, 2015, at 10:03 PM, PRANEESH KUMAR <pr...@gmail.com> wrote:
>
> Hi Michael,
>
> We can have a monitoring thread and interrupt the hbase client thread
> after time out instead of doing this I want the timeout or some exception
> to be thrown from the HBase client itself.
>
> On Thu, Jun 11, 2015 at 5:16 AM, Michael Segel <mi...@hotmail.com>
> wrote:
>
>> threads?
>>
>> So that regardless of your hadoop settings, if you want something faster,
>> you can use one thread for a timer and then the request is in another. So
>> if you hit your timeout before you get a response, you can stop your thread.
>> (YMMV depending on side effects… )
>>
>>> On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR <pr...@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> I have got the Connection object with default configuration, if the
>>> zookeeper or HMaster or Region server is down, the client didn't fast
>> fail
>>> and it took almost 20 mins to thrown an error.
>>>
>>> What is the best configuration to make the client fast fail.
>>>
>>> Also what is significance of changing the following parameters.
>>>
>>> hbase.client.retries.number
>>> zookeeper.recovery.retry
>>> zookeeper.session.timeout
>>> zookeeper.recovery.retry.intervalmill
>>> hbase.rpc.timeout
>>>
>>> Regards,
>>> Praneesh
>>
>>
Re: How to make the client fast fail
Posted by lars hofhansl <la...@apache.org>.
Filed https://issues.apache.org/jira/browse/HBASE-13919, please chime in there.
Thanks.
-- Lars
From: lars hofhansl <la...@apache.org>
To: mukund murrali <mu...@gmail.com>; "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Tuesday, June 16, 2015 12:37 PM
Subject: Re: How to make the client fast fail
I agree. We have to make this easier - to configure and to understand. And the default should be better. Looks like they only make sense for long running M/R jobs.
Lemme check whether there's a jira already; I'll revive it, otherwise I'll create one and we can have further discussions there.
Thanks for pointing this issue out.
-- Lars
From: mukund murrali <mu...@gmail.com>
To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
Sent: Tuesday, June 16, 2015 12:21 AM
Subject: Re: How to make the client fast fail
We are using HBase - 1.0.0. Yes we have went through this blog. But configuring these parameters, we not able to find out what is the exact time it takes to fail fast. I am really curious if there could be a single configuration to ensure client level failing? Also it would be great if someone can suggest some optimal value for those parameters.
On Tue, Jun 16, 2015 at 12:43 PM, lars hofhansl <la...@apache.org> wrote:
Please always tell us which version of HBase you are using. We have fixed a lot of issues in this area over time.Here's an _old_ blog post I wrote about this: http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html
Using yet more threads to monitor timeouts of another thread is a bad idea, especially when the timeout is configurable in the first place.
-- Lars
From: mukund murrali <mu...@gmail.com>
To: user@hbase.apache.org
Sent: Sunday, June 14, 2015 10:22 PM
Subject: Re: How to make the client fast fail
It would be great if there is a single timeout configuration from the
client end. All other parameters should fine tune based on that one
parameter. We have modified simple based on trail basis to suit our need.
Also not sure what side effect it would cause configuring those parameters.
On Mon, Jun 15, 2015 at 10:38 AM, <Ha...@dell.com> wrote:
> We are also interested on the solution for this. With
> hbase.client.retries.number = 7 and client.pause=400ms, it came down to
> ~9mins (from 20 mins). Now we are thinking the 9mins is also a big number.
>
> Thanks,
> Hari
>
> -----Original Message-----
> From: PRANEESH KUMAR [mailto:praneesh.sankar@gmail.com]
> Sent: Monday, June 15, 2015 10:33 AM
> To: user@hbase.apache.org
> Subject: Re: How to make the client fast fail
>
> Hi Michael,
>
> We can have a monitoring thread and interrupt the hbase client thread
> after time out instead of doing this I want the timeout or some exception
> to be thrown from the HBase client itself.
>
> On Thu, Jun 11, 2015 at 5:16 AM, Michael Segel
> wrote:
>
> > threads?
> >
> > So that regardless of your hadoop settings, if you want something
> > faster, you can use one thread for a timer and then the request is in
> > another. So if you hit your timeout before you get a response, you can
> stop your thread.
> > (YMMV depending on side effects... )
> >
> > > On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR
> > >
> > wrote:
> > >
> > > Hi,
> > >
> > > I have got the Connection object with default configuration, if the
> > > zookeeper or HMaster or Region server is down, the client didn't
> > > fast
> > fail
> > > and it took almost 20 mins to thrown an error.
> > >
> > > What is the best configuration to make the client fast fail.
> > >
> > > Also what is significance of changing the following parameters.
> > >
> > > hbase.client.retries.number
> > > zookeeper.recovery.retry
> > > zookeeper.session.timeout
> > > zookeeper.recovery.retry.intervalmill
> > > hbase.rpc.timeout
> > >
> > > Regards,
> > > Praneesh
> >
> >
>
Re: How to make the client fast fail
Posted by lars hofhansl <la...@apache.org>.
I agree. We have to make this easier - to configure and to understand. And the default should be better. Looks like they only make sense for long running M/R jobs.
Lemme check whether there's a jira already; I'll revive it, otherwise I'll create one and we can have further discussions there.
Thanks for pointing this issue out.
-- Lars
From: mukund murrali <mu...@gmail.com>
To: user@hbase.apache.org; lars hofhansl <la...@apache.org>
Sent: Tuesday, June 16, 2015 12:21 AM
Subject: Re: How to make the client fast fail
We are using HBase - 1.0.0. Yes we have went through this blog. But configuring these parameters, we not able to find out what is the exact time it takes to fail fast. I am really curious if there could be a single configuration to ensure client level failing? Also it would be great if someone can suggest some optimal value for those parameters.
On Tue, Jun 16, 2015 at 12:43 PM, lars hofhansl <la...@apache.org> wrote:
Please always tell us which version of HBase you are using. We have fixed a lot of issues in this area over time.Here's an _old_ blog post I wrote about this: http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html
Using yet more threads to monitor timeouts of another thread is a bad idea, especially when the timeout is configurable in the first place.
-- Lars
From: mukund murrali <mu...@gmail.com>
To: user@hbase.apache.org
Sent: Sunday, June 14, 2015 10:22 PM
Subject: Re: How to make the client fast fail
It would be great if there is a single timeout configuration from the
client end. All other parameters should fine tune based on that one
parameter. We have modified simple based on trail basis to suit our need.
Also not sure what side effect it would cause configuring those parameters.
On Mon, Jun 15, 2015 at 10:38 AM, <Ha...@dell.com> wrote:
> We are also interested on the solution for this. With
> hbase.client.retries.number = 7 and client.pause=400ms, it came down to
> ~9mins (from 20 mins). Now we are thinking the 9mins is also a big number.
>
> Thanks,
> Hari
>
> -----Original Message-----
> From: PRANEESH KUMAR [mailto:praneesh.sankar@gmail.com]
> Sent: Monday, June 15, 2015 10:33 AM
> To: user@hbase.apache.org
> Subject: Re: How to make the client fast fail
>
> Hi Michael,
>
> We can have a monitoring thread and interrupt the hbase client thread
> after time out instead of doing this I want the timeout or some exception
> to be thrown from the HBase client itself.
>
> On Thu, Jun 11, 2015 at 5:16 AM, Michael Segel
> wrote:
>
> > threads?
> >
> > So that regardless of your hadoop settings, if you want something
> > faster, you can use one thread for a timer and then the request is in
> > another. So if you hit your timeout before you get a response, you can
> stop your thread.
> > (YMMV depending on side effects... )
> >
> > > On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR
> > >
> > wrote:
> > >
> > > Hi,
> > >
> > > I have got the Connection object with default configuration, if the
> > > zookeeper or HMaster or Region server is down, the client didn't
> > > fast
> > fail
> > > and it took almost 20 mins to thrown an error.
> > >
> > > What is the best configuration to make the client fast fail.
> > >
> > > Also what is significance of changing the following parameters.
> > >
> > > hbase.client.retries.number
> > > zookeeper.recovery.retry
> > > zookeeper.session.timeout
> > > zookeeper.recovery.retry.intervalmill
> > > hbase.rpc.timeout
> > >
> > > Regards,
> > > Praneesh
> >
> >
>
RE: How to make the client fast fail
Posted by Ha...@Dell.com.
In addition to 1.0.0, also suggest for 0.94.15.
Thanks,
Hari
-----Original Message-----
From: mukund murrali [mailto:mukundmurrali9@gmail.com]
Sent: Tuesday, June 16, 2015 12:52 PM
To: user@hbase.apache.org; lars hofhansl
Subject: Re: How to make the client fast fail
We are using HBase - 1.0.0. Yes we have went through this blog. But configuring these parameters, we not able to find out what is the exact time it takes to fail fast. I am really curious if there could be a single configuration to ensure client level failing? Also it would be great if someone can suggest some optimal value for those parameters.
On Tue, Jun 16, 2015 at 12:43 PM, lars hofhansl wrote:
> Please always tell us which version of HBase you are using. We have
> fixed a lot of issues in this area over time.Here's an _old_ blog post
> I wrote about this:
> http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html
>
> Using yet more threads to monitor timeouts of another thread is a bad
> idea, especially when the timeout is configurable in the first place.
>
> -- Lars
> From: mukund murrali
> To: user@hbase.apache.org
> Sent: Sunday, June 14, 2015 10:22 PM
> Subject: Re: How to make the client fast fail
>
> It would be great if there is a single timeout configuration from the
> client end. All other parameters should fine tune based on that one
> parameter. We have modified simple based on trail basis to suit our need.
> Also not sure what side effect it would cause configuring those parameters.
>
>
>
> On Mon, Jun 15, 2015 at 10:38 AM, wrote:
>
> > We are also interested on the solution for this. With
> > hbase.client.retries.number = 7 and client.pause=400ms, it came down
> > to ~9mins (from 20 mins). Now we are thinking the 9mins is also a
> > big
> number.
> >
> > Thanks,
> > Hari
> >
> > -----Original Message-----
> > From: PRANEESH KUMAR [mailto:praneesh.sankar@gmail.com]
> > Sent: Monday, June 15, 2015 10:33 AM
> > To: user@hbase.apache.org
> > Subject: Re: How to make the client fast fail
> >
> > Hi Michael,
> >
> > We can have a monitoring thread and interrupt the hbase client
> > thread after time out instead of doing this I want the timeout or
> > some exception to be thrown from the HBase client itself.
> >
> > On Thu, Jun 11, 2015 at 5:16 AM, Michael Segel
> > wrote:
> >
> > > threads?
> > >
> > > So that regardless of your hadoop settings, if you want something
> > > faster, you can use one thread for a timer and then the request is
> > > in another. So if you hit your timeout before you get a response,
> > > you can
> > stop your thread.
> > > (YMMV depending on side effects... )
> > >
> > > > On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR
> > > >
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > I have got the Connection object with default configuration, if
> > > > the zookeeper or HMaster or Region server is down, the client
> > > > didn't fast
> > > fail
> > > > and it took almost 20 mins to thrown an error.
> > > >
> > > > What is the best configuration to make the client fast fail.
> > > >
> > > > Also what is significance of changing the following parameters.
> > > >
> > > > hbase.client.retries.number
> > > > zookeeper.recovery.retry
> > > > zookeeper.session.timeout
> > > > zookeeper.recovery.retry.intervalmill
> > > > hbase.rpc.timeout
> > > >
> > > > Regards,
> > > > Praneesh
> > >
> > >
> >
>
>
>
>
Re: How to make the client fast fail
Posted by mukund murrali <mu...@gmail.com>.
We are using HBase - 1.0.0. Yes we have went through this blog. But
configuring these parameters, we not able to find out what is the exact
time it takes to fail fast. I am really curious if there could be a single
configuration to ensure client level failing? Also it would be great if
someone can suggest some optimal value for those parameters.
On Tue, Jun 16, 2015 at 12:43 PM, lars hofhansl <la...@apache.org> wrote:
> Please always tell us which version of HBase you are using. We have fixed
> a lot of issues in this area over time.Here's an _old_ blog post I wrote
> about this:
> http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html
>
> Using yet more threads to monitor timeouts of another thread is a bad
> idea, especially when the timeout is configurable in the first place.
>
> -- Lars
> From: mukund murrali <mu...@gmail.com>
> To: user@hbase.apache.org
> Sent: Sunday, June 14, 2015 10:22 PM
> Subject: Re: How to make the client fast fail
>
> It would be great if there is a single timeout configuration from the
> client end. All other parameters should fine tune based on that one
> parameter. We have modified simple based on trail basis to suit our need.
> Also not sure what side effect it would cause configuring those parameters.
>
>
>
> On Mon, Jun 15, 2015 at 10:38 AM, <Ha...@dell.com> wrote:
>
> > We are also interested on the solution for this. With
> > hbase.client.retries.number = 7 and client.pause=400ms, it came down to
> > ~9mins (from 20 mins). Now we are thinking the 9mins is also a big
> number.
> >
> > Thanks,
> > Hari
> >
> > -----Original Message-----
> > From: PRANEESH KUMAR [mailto:praneesh.sankar@gmail.com]
> > Sent: Monday, June 15, 2015 10:33 AM
> > To: user@hbase.apache.org
> > Subject: Re: How to make the client fast fail
> >
> > Hi Michael,
> >
> > We can have a monitoring thread and interrupt the hbase client thread
> > after time out instead of doing this I want the timeout or some exception
> > to be thrown from the HBase client itself.
> >
> > On Thu, Jun 11, 2015 at 5:16 AM, Michael Segel
> > wrote:
> >
> > > threads?
> > >
> > > So that regardless of your hadoop settings, if you want something
> > > faster, you can use one thread for a timer and then the request is in
> > > another. So if you hit your timeout before you get a response, you can
> > stop your thread.
> > > (YMMV depending on side effects... )
> > >
> > > > On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR
> > > >
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > I have got the Connection object with default configuration, if the
> > > > zookeeper or HMaster or Region server is down, the client didn't
> > > > fast
> > > fail
> > > > and it took almost 20 mins to thrown an error.
> > > >
> > > > What is the best configuration to make the client fast fail.
> > > >
> > > > Also what is significance of changing the following parameters.
> > > >
> > > > hbase.client.retries.number
> > > > zookeeper.recovery.retry
> > > > zookeeper.session.timeout
> > > > zookeeper.recovery.retry.intervalmill
> > > > hbase.rpc.timeout
> > > >
> > > > Regards,
> > > > Praneesh
> > >
> > >
> >
>
>
>
>
Re: How to make the client fast fail
Posted by mukund murrali <mu...@gmail.com>.
As for now, any suggestions what values those configurations can have.
Thanks
On Mon, Jun 22, 2015 at 11:52 PM, Michael Segel <mi...@hotmail.com>
wrote:
> Uhm… what happens when you hit a parameter that was made FINAL?
>
> ;-)
>
> Yes, I agree that you can change some of the parameters at the application
> level.
> Using a timer thread is just as easy and you don’t have to worry about the
> fact that your admins made certain parameters final in their configuration
> because of KISS.
>
> There are other reasons why you may want to use a timer thread… It really
> depends on what you want to do and why.
>
> > On Jun 16, 2015, at 1:08 PM, Bryan Beaudreault <bb...@hubspot.com>
> wrote:
> >
> > I agree that more documentation would be better. However,
> >
> >
> >> Yet, there are some applications which require a faster time out than
> >> others. So, you tune some of the timers to have a fast fail, and you
> end up
> >> causing unintended problems for others.
> >>
> >> The simplest solution is to use threads in you client app. (Of course
> this
> >> assumes that you’re capable of writing clean multi-threaded code and I
> >> don’t want to assume anything.)
> >>
> >> Remember that HBase is a shared resource. So you need to consider the
> >> whole at the same time you consider the needs of one.
> >>
> >>
> > This is not really true, it assumes a very naive configuration solution
> in
> > which all processes are configured the same. The default configs come
> from
> > xml files, but are easily customized. Consider:
> >
> > Application A:
> >
> > Configuration conf = HBaseConfiguration.create();
> > conf.setInt("hbase.rpc.timeout", 8000);
> >
> > // Operations on table have a timeout of 8s
> > Connection conn = ConnectionFactory.createConnection(conf);
> > conn.getTable("foo");
> >
> > Application B:
> >
> > // Operations on this table use default timeout
> > Connection conn =
> > ConnectionFactory.createConnection(HBaseConfiguration.create());
> > conn.getTable("foo");
> >
> > // Operations on this table use a very short timeout of 500ms
> > Configuration conf = HBaseConfiguration.create();
> > conf.setInt("hbase.rpc.timeout", 500);
> > Connection shortConn = ConnectionFactory.createConnection(conf);
> > short
> > shortConn.getTable("foo");
> >
> > Applications A and B are configured with different timeouts. Further,
> > Application B has two separate table connections, each with a different
> > timeout.
> >
> > The values are hardcoded above, but could easily be made configurable.
> The
> > simplest of solutions uses System.getProperty(), so you can pass
> > -Dmy.custom.timeout=500. More complex solutions can utilize the various
> > open source live configuration solutions. Such as netflix archaius, or
> > hubspot's live-config -- both available on github.
> >
> > Of course there can be unintended consequences if an application suddenly
> >> starts to drop connections before a result or timeout occurs too. ;-)
> >>
> >>
> >>> On Jun 16, 2015, at 12:13 AM, lars hofhansl <la...@apache.org> wrote:
> >>>
> >>> Please always tell us which version of HBase you are using. We have
> >> fixed a lot of issues in this area over time.Here's an _old_ blog post I
> >> wrote about this:
> >> http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html
> >>>
> >>> Using yet more threads to monitor timeouts of another thread is a bad
> >> idea, especially when the timeout is configurable in the first place.
> >>>
> >>> -- Lars
> >>> From: mukund murrali <mu...@gmail.com>
> >>> To: user@hbase.apache.org
> >>> Sent: Sunday, June 14, 2015 10:22 PM
> >>> Subject: Re: How to make the client fast fail
> >>>
> >>> It would be great if there is a single timeout configuration from the
> >>> client end. All other parameters should fine tune based on that one
> >>> parameter. We have modified simple based on trail basis to suit our
> need.
> >>> Also not sure what side effect it would cause configuring those
> >> parameters.
> >>>
> >>>
> >>>
> >>> On Mon, Jun 15, 2015 at 10:38 AM, <Ha...@dell.com>
> wrote:
> >>>
> >>>> We are also interested on the solution for this. With
> >>>> hbase.client.retries.number = 7 and client.pause=400ms, it came down
> to
> >>>> ~9mins (from 20 mins). Now we are thinking the 9mins is also a big
> >> number.
> >>>>
> >>>> Thanks,
> >>>> Hari
> >>>>
> >>>> -----Original Message-----
> >>>> From: PRANEESH KUMAR [mailto:praneesh.sankar@gmail.com]
> >>>> Sent: Monday, June 15, 2015 10:33 AM
> >>>> To: user@hbase.apache.org
> >>>> Subject: Re: How to make the client fast fail
> >>>>
> >>>> Hi Michael,
> >>>>
> >>>> We can have a monitoring thread and interrupt the hbase client thread
> >>>> after time out instead of doing this I want the timeout or some
> >> exception
> >>>> to be thrown from the HBase client itself.
> >>>>
> >>>> On Thu, Jun 11, 2015 at 5:16 AM, Michael Segel
> >>>> wrote:
> >>>>
> >>>>> threads?
> >>>>>
> >>>>> So that regardless of your hadoop settings, if you want something
> >>>>> faster, you can use one thread for a timer and then the request is in
> >>>>> another. So if you hit your timeout before you get a response, you
> can
> >>>> stop your thread.
> >>>>> (YMMV depending on side effects... )
> >>>>>
> >>>>>> On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR
> >>>>>>
> >>>>> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I have got the Connection object with default configuration, if the
> >>>>>> zookeeper or HMaster or Region server is down, the client didn't
> >>>>>> fast
> >>>>> fail
> >>>>>> and it took almost 20 mins to thrown an error.
> >>>>>>
> >>>>>> What is the best configuration to make the client fast fail.
> >>>>>>
> >>>>>> Also what is significance of changing the following parameters.
> >>>>>>
> >>>>>> hbase.client.retries.number
> >>>>>> zookeeper.recovery.retry
> >>>>>> zookeeper.session.timeout
> >>>>>> zookeeper.recovery.retry.intervalmill
> >>>>>> hbase.rpc.timeout
> >>>>>>
> >>>>>> Regards,
> >>>>>> Praneesh
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>
> >>
>
> The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>
Re: How to make the client fast fail
Posted by Michael Segel <mi...@hotmail.com>.
Uhm… what happens when you hit a parameter that was made FINAL?
;-)
Yes, I agree that you can change some of the parameters at the application level.
Using a timer thread is just as easy and you don’t have to worry about the fact that your admins made certain parameters final in their configuration because of KISS.
There are other reasons why you may want to use a timer thread… It really depends on what you want to do and why.
> On Jun 16, 2015, at 1:08 PM, Bryan Beaudreault <bb...@hubspot.com> wrote:
>
> I agree that more documentation would be better. However,
>
>
>> Yet, there are some applications which require a faster time out than
>> others. So, you tune some of the timers to have a fast fail, and you end up
>> causing unintended problems for others.
>>
>> The simplest solution is to use threads in you client app. (Of course this
>> assumes that you’re capable of writing clean multi-threaded code and I
>> don’t want to assume anything.)
>>
>> Remember that HBase is a shared resource. So you need to consider the
>> whole at the same time you consider the needs of one.
>>
>>
> This is not really true, it assumes a very naive configuration solution in
> which all processes are configured the same. The default configs come from
> xml files, but are easily customized. Consider:
>
> Application A:
>
> Configuration conf = HBaseConfiguration.create();
> conf.setInt("hbase.rpc.timeout", 8000);
>
> // Operations on table have a timeout of 8s
> Connection conn = ConnectionFactory.createConnection(conf);
> conn.getTable("foo");
>
> Application B:
>
> // Operations on this table use default timeout
> Connection conn =
> ConnectionFactory.createConnection(HBaseConfiguration.create());
> conn.getTable("foo");
>
> // Operations on this table use a very short timeout of 500ms
> Configuration conf = HBaseConfiguration.create();
> conf.setInt("hbase.rpc.timeout", 500);
> Connection shortConn = ConnectionFactory.createConnection(conf);
> short
> shortConn.getTable("foo");
>
> Applications A and B are configured with different timeouts. Further,
> Application B has two separate table connections, each with a different
> timeout.
>
> The values are hardcoded above, but could easily be made configurable. The
> simplest of solutions uses System.getProperty(), so you can pass
> -Dmy.custom.timeout=500. More complex solutions can utilize the various
> open source live configuration solutions. Such as netflix archaius, or
> hubspot's live-config -- both available on github.
>
> Of course there can be unintended consequences if an application suddenly
>> starts to drop connections before a result or timeout occurs too. ;-)
>>
>>
>>> On Jun 16, 2015, at 12:13 AM, lars hofhansl <la...@apache.org> wrote:
>>>
>>> Please always tell us which version of HBase you are using. We have
>> fixed a lot of issues in this area over time.Here's an _old_ blog post I
>> wrote about this:
>> http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html
>>>
>>> Using yet more threads to monitor timeouts of another thread is a bad
>> idea, especially when the timeout is configurable in the first place.
>>>
>>> -- Lars
>>> From: mukund murrali <mu...@gmail.com>
>>> To: user@hbase.apache.org
>>> Sent: Sunday, June 14, 2015 10:22 PM
>>> Subject: Re: How to make the client fast fail
>>>
>>> It would be great if there is a single timeout configuration from the
>>> client end. All other parameters should fine tune based on that one
>>> parameter. We have modified simple based on trail basis to suit our need.
>>> Also not sure what side effect it would cause configuring those
>> parameters.
>>>
>>>
>>>
>>> On Mon, Jun 15, 2015 at 10:38 AM, <Ha...@dell.com> wrote:
>>>
>>>> We are also interested on the solution for this. With
>>>> hbase.client.retries.number = 7 and client.pause=400ms, it came down to
>>>> ~9mins (from 20 mins). Now we are thinking the 9mins is also a big
>> number.
>>>>
>>>> Thanks,
>>>> Hari
>>>>
>>>> -----Original Message-----
>>>> From: PRANEESH KUMAR [mailto:praneesh.sankar@gmail.com]
>>>> Sent: Monday, June 15, 2015 10:33 AM
>>>> To: user@hbase.apache.org
>>>> Subject: Re: How to make the client fast fail
>>>>
>>>> Hi Michael,
>>>>
>>>> We can have a monitoring thread and interrupt the hbase client thread
>>>> after time out instead of doing this I want the timeout or some
>> exception
>>>> to be thrown from the HBase client itself.
>>>>
>>>> On Thu, Jun 11, 2015 at 5:16 AM, Michael Segel
>>>> wrote:
>>>>
>>>>> threads?
>>>>>
>>>>> So that regardless of your hadoop settings, if you want something
>>>>> faster, you can use one thread for a timer and then the request is in
>>>>> another. So if you hit your timeout before you get a response, you can
>>>> stop your thread.
>>>>> (YMMV depending on side effects... )
>>>>>
>>>>>> On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR
>>>>>>
>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have got the Connection object with default configuration, if the
>>>>>> zookeeper or HMaster or Region server is down, the client didn't
>>>>>> fast
>>>>> fail
>>>>>> and it took almost 20 mins to thrown an error.
>>>>>>
>>>>>> What is the best configuration to make the client fast fail.
>>>>>>
>>>>>> Also what is significance of changing the following parameters.
>>>>>>
>>>>>> hbase.client.retries.number
>>>>>> zookeeper.recovery.retry
>>>>>> zookeeper.session.timeout
>>>>>> zookeeper.recovery.retry.intervalmill
>>>>>> hbase.rpc.timeout
>>>>>>
>>>>>> Regards,
>>>>>> Praneesh
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: How to make the client fast fail
Posted by Bryan Beaudreault <bb...@hubspot.com>.
I agree that more documentation would be better. However,
> Yet, there are some applications which require a faster time out than
> others. So, you tune some of the timers to have a fast fail, and you end up
> causing unintended problems for others.
>
> The simplest solution is to use threads in you client app. (Of course this
> assumes that you’re capable of writing clean multi-threaded code and I
> don’t want to assume anything.)
>
> Remember that HBase is a shared resource. So you need to consider the
> whole at the same time you consider the needs of one.
>
>
This is not really true, it assumes a very naive configuration solution in
which all processes are configured the same. The default configs come from
xml files, but are easily customized. Consider:
Application A:
Configuration conf = HBaseConfiguration.create();
conf.setInt("hbase.rpc.timeout", 8000);
// Operations on table have a timeout of 8s
Connection conn = ConnectionFactory.createConnection(conf);
conn.getTable("foo");
Application B:
// Operations on this table use default timeout
Connection conn =
ConnectionFactory.createConnection(HBaseConfiguration.create());
conn.getTable("foo");
// Operations on this table use a very short timeout of 500ms
Configuration conf = HBaseConfiguration.create();
conf.setInt("hbase.rpc.timeout", 500);
Connection shortConn = ConnectionFactory.createConnection(conf);
short
shortConn.getTable("foo");
Applications A and B are configured with different timeouts. Further,
Application B has two separate table connections, each with a different
timeout.
The values are hardcoded above, but could easily be made configurable. The
simplest of solutions uses System.getProperty(), so you can pass
-Dmy.custom.timeout=500. More complex solutions can utilize the various
open source live configuration solutions. Such as netflix archaius, or
hubspot's live-config -- both available on github.
Of course there can be unintended consequences if an application suddenly
> starts to drop connections before a result or timeout occurs too. ;-)
>
>
> > On Jun 16, 2015, at 12:13 AM, lars hofhansl <la...@apache.org> wrote:
> >
> > Please always tell us which version of HBase you are using. We have
> fixed a lot of issues in this area over time.Here's an _old_ blog post I
> wrote about this:
> http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html
> >
> > Using yet more threads to monitor timeouts of another thread is a bad
> idea, especially when the timeout is configurable in the first place.
> >
> > -- Lars
> > From: mukund murrali <mu...@gmail.com>
> > To: user@hbase.apache.org
> > Sent: Sunday, June 14, 2015 10:22 PM
> > Subject: Re: How to make the client fast fail
> >
> > It would be great if there is a single timeout configuration from the
> > client end. All other parameters should fine tune based on that one
> > parameter. We have modified simple based on trail basis to suit our need.
> > Also not sure what side effect it would cause configuring those
> parameters.
> >
> >
> >
> > On Mon, Jun 15, 2015 at 10:38 AM, <Ha...@dell.com> wrote:
> >
> >> We are also interested on the solution for this. With
> >> hbase.client.retries.number = 7 and client.pause=400ms, it came down to
> >> ~9mins (from 20 mins). Now we are thinking the 9mins is also a big
> number.
> >>
> >> Thanks,
> >> Hari
> >>
> >> -----Original Message-----
> >> From: PRANEESH KUMAR [mailto:praneesh.sankar@gmail.com]
> >> Sent: Monday, June 15, 2015 10:33 AM
> >> To: user@hbase.apache.org
> >> Subject: Re: How to make the client fast fail
> >>
> >> Hi Michael,
> >>
> >> We can have a monitoring thread and interrupt the hbase client thread
> >> after time out instead of doing this I want the timeout or some
> exception
> >> to be thrown from the HBase client itself.
> >>
> >> On Thu, Jun 11, 2015 at 5:16 AM, Michael Segel
> >> wrote:
> >>
> >>> threads?
> >>>
> >>> So that regardless of your hadoop settings, if you want something
> >>> faster, you can use one thread for a timer and then the request is in
> >>> another. So if you hit your timeout before you get a response, you can
> >> stop your thread.
> >>> (YMMV depending on side effects... )
> >>>
> >>>> On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR
> >>>>
> >>> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> I have got the Connection object with default configuration, if the
> >>>> zookeeper or HMaster or Region server is down, the client didn't
> >>>> fast
> >>> fail
> >>>> and it took almost 20 mins to thrown an error.
> >>>>
> >>>> What is the best configuration to make the client fast fail.
> >>>>
> >>>> Also what is significance of changing the following parameters.
> >>>>
> >>>> hbase.client.retries.number
> >>>> zookeeper.recovery.retry
> >>>> zookeeper.session.timeout
> >>>> zookeeper.recovery.retry.intervalmill
> >>>> hbase.rpc.timeout
> >>>>
> >>>> Regards,
> >>>> Praneesh
> >>>
> >>>
> >>
> >
> >
>
>
Re: How to make the client fast fail
Posted by Michael Segel <mi...@hotmail.com>.
Lars,
Sigh.
Yes, configuring your timeouts correctly is important.
Time is very important in distributed systems.
Yet, there are some applications which require a faster time out than others. So, you tune some of the timers to have a fast fail, and you end up causing unintended problems for others.
The simplest solution is to use threads in you client app. (Of course this assumes that you’re capable of writing clean multi-threaded code and I don’t want to assume anything.)
Remember that HBase is a shared resource. So you need to consider the whole at the same time you consider the needs of one.
Of course there can be unintended consequences if an application suddenly starts to drop connections before a result or timeout occurs too. ;-)
> On Jun 16, 2015, at 12:13 AM, lars hofhansl <la...@apache.org> wrote:
>
> Please always tell us which version of HBase you are using. We have fixed a lot of issues in this area over time.Here's an _old_ blog post I wrote about this: http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html
>
> Using yet more threads to monitor timeouts of another thread is a bad idea, especially when the timeout is configurable in the first place.
>
> -- Lars
> From: mukund murrali <mu...@gmail.com>
> To: user@hbase.apache.org
> Sent: Sunday, June 14, 2015 10:22 PM
> Subject: Re: How to make the client fast fail
>
> It would be great if there is a single timeout configuration from the
> client end. All other parameters should fine tune based on that one
> parameter. We have modified simple based on trail basis to suit our need.
> Also not sure what side effect it would cause configuring those parameters.
>
>
>
> On Mon, Jun 15, 2015 at 10:38 AM, <Ha...@dell.com> wrote:
>
>> We are also interested on the solution for this. With
>> hbase.client.retries.number = 7 and client.pause=400ms, it came down to
>> ~9mins (from 20 mins). Now we are thinking the 9mins is also a big number.
>>
>> Thanks,
>> Hari
>>
>> -----Original Message-----
>> From: PRANEESH KUMAR [mailto:praneesh.sankar@gmail.com]
>> Sent: Monday, June 15, 2015 10:33 AM
>> To: user@hbase.apache.org
>> Subject: Re: How to make the client fast fail
>>
>> Hi Michael,
>>
>> We can have a monitoring thread and interrupt the hbase client thread
>> after time out instead of doing this I want the timeout or some exception
>> to be thrown from the HBase client itself.
>>
>> On Thu, Jun 11, 2015 at 5:16 AM, Michael Segel
>> wrote:
>>
>>> threads?
>>>
>>> So that regardless of your hadoop settings, if you want something
>>> faster, you can use one thread for a timer and then the request is in
>>> another. So if you hit your timeout before you get a response, you can
>> stop your thread.
>>> (YMMV depending on side effects... )
>>>
>>>> On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR
>>>>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have got the Connection object with default configuration, if the
>>>> zookeeper or HMaster or Region server is down, the client didn't
>>>> fast
>>> fail
>>>> and it took almost 20 mins to thrown an error.
>>>>
>>>> What is the best configuration to make the client fast fail.
>>>>
>>>> Also what is significance of changing the following parameters.
>>>>
>>>> hbase.client.retries.number
>>>> zookeeper.recovery.retry
>>>> zookeeper.session.timeout
>>>> zookeeper.recovery.retry.intervalmill
>>>> hbase.rpc.timeout
>>>>
>>>> Regards,
>>>> Praneesh
>>>
>>>
>>
>
>
Re: How to make the client fast fail
Posted by lars hofhansl <la...@apache.org>.
Please always tell us which version of HBase you are using. We have fixed a lot of issues in this area over time.Here's an _old_ blog post I wrote about this: http://hadoop-hbase.blogspot.com/2012/09/hbase-client-timeouts.html
Using yet more threads to monitor timeouts of another thread is a bad idea, especially when the timeout is configurable in the first place.
-- Lars
From: mukund murrali <mu...@gmail.com>
To: user@hbase.apache.org
Sent: Sunday, June 14, 2015 10:22 PM
Subject: Re: How to make the client fast fail
It would be great if there is a single timeout configuration from the
client end. All other parameters should fine tune based on that one
parameter. We have modified simple based on trail basis to suit our need.
Also not sure what side effect it would cause configuring those parameters.
On Mon, Jun 15, 2015 at 10:38 AM, <Ha...@dell.com> wrote:
> We are also interested on the solution for this. With
> hbase.client.retries.number = 7 and client.pause=400ms, it came down to
> ~9mins (from 20 mins). Now we are thinking the 9mins is also a big number.
>
> Thanks,
> Hari
>
> -----Original Message-----
> From: PRANEESH KUMAR [mailto:praneesh.sankar@gmail.com]
> Sent: Monday, June 15, 2015 10:33 AM
> To: user@hbase.apache.org
> Subject: Re: How to make the client fast fail
>
> Hi Michael,
>
> We can have a monitoring thread and interrupt the hbase client thread
> after time out instead of doing this I want the timeout or some exception
> to be thrown from the HBase client itself.
>
> On Thu, Jun 11, 2015 at 5:16 AM, Michael Segel
> wrote:
>
> > threads?
> >
> > So that regardless of your hadoop settings, if you want something
> > faster, you can use one thread for a timer and then the request is in
> > another. So if you hit your timeout before you get a response, you can
> stop your thread.
> > (YMMV depending on side effects... )
> >
> > > On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR
> > >
> > wrote:
> > >
> > > Hi,
> > >
> > > I have got the Connection object with default configuration, if the
> > > zookeeper or HMaster or Region server is down, the client didn't
> > > fast
> > fail
> > > and it took almost 20 mins to thrown an error.
> > >
> > > What is the best configuration to make the client fast fail.
> > >
> > > Also what is significance of changing the following parameters.
> > >
> > > hbase.client.retries.number
> > > zookeeper.recovery.retry
> > > zookeeper.session.timeout
> > > zookeeper.recovery.retry.intervalmill
> > > hbase.rpc.timeout
> > >
> > > Regards,
> > > Praneesh
> >
> >
>
Re: How to make the client fast fail
Posted by mukund murrali <mu...@gmail.com>.
It would be great if there is a single timeout configuration from the
client end. All other parameters should fine tune based on that one
parameter. We have modified simple based on trail basis to suit our need.
Also not sure what side effect it would cause configuring those parameters.
On Mon, Jun 15, 2015 at 10:38 AM, <Ha...@dell.com> wrote:
> We are also interested on the solution for this. With
> hbase.client.retries.number = 7 and client.pause=400ms, it came down to
> ~9mins (from 20 mins). Now we are thinking the 9mins is also a big number.
>
> Thanks,
> Hari
>
> -----Original Message-----
> From: PRANEESH KUMAR [mailto:praneesh.sankar@gmail.com]
> Sent: Monday, June 15, 2015 10:33 AM
> To: user@hbase.apache.org
> Subject: Re: How to make the client fast fail
>
> Hi Michael,
>
> We can have a monitoring thread and interrupt the hbase client thread
> after time out instead of doing this I want the timeout or some exception
> to be thrown from the HBase client itself.
>
> On Thu, Jun 11, 2015 at 5:16 AM, Michael Segel
> wrote:
>
> > threads?
> >
> > So that regardless of your hadoop settings, if you want something
> > faster, you can use one thread for a timer and then the request is in
> > another. So if you hit your timeout before you get a response, you can
> stop your thread.
> > (YMMV depending on side effects... )
> >
> > > On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR
> > >
> > wrote:
> > >
> > > Hi,
> > >
> > > I have got the Connection object with default configuration, if the
> > > zookeeper or HMaster or Region server is down, the client didn't
> > > fast
> > fail
> > > and it took almost 20 mins to thrown an error.
> > >
> > > What is the best configuration to make the client fast fail.
> > >
> > > Also what is significance of changing the following parameters.
> > >
> > > hbase.client.retries.number
> > > zookeeper.recovery.retry
> > > zookeeper.session.timeout
> > > zookeeper.recovery.retry.intervalmill
> > > hbase.rpc.timeout
> > >
> > > Regards,
> > > Praneesh
> >
> >
>
RE: How to make the client fast fail
Posted by Ha...@Dell.com.
We are also interested on the solution for this. With hbase.client.retries.number = 7 and client.pause=400ms, it came down to ~9mins (from 20 mins). Now we are thinking the 9mins is also a big number.
Thanks,
Hari
-----Original Message-----
From: PRANEESH KUMAR [mailto:praneesh.sankar@gmail.com]
Sent: Monday, June 15, 2015 10:33 AM
To: user@hbase.apache.org
Subject: Re: How to make the client fast fail
Hi Michael,
We can have a monitoring thread and interrupt the hbase client thread after time out instead of doing this I want the timeout or some exception to be thrown from the HBase client itself.
On Thu, Jun 11, 2015 at 5:16 AM, Michael Segel
wrote:
> threads?
>
> So that regardless of your hadoop settings, if you want something
> faster, you can use one thread for a timer and then the request is in
> another. So if you hit your timeout before you get a response, you can stop your thread.
> (YMMV depending on side effects... )
>
> > On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR
> >
> wrote:
> >
> > Hi,
> >
> > I have got the Connection object with default configuration, if the
> > zookeeper or HMaster or Region server is down, the client didn't
> > fast
> fail
> > and it took almost 20 mins to thrown an error.
> >
> > What is the best configuration to make the client fast fail.
> >
> > Also what is significance of changing the following parameters.
> >
> > hbase.client.retries.number
> > zookeeper.recovery.retry
> > zookeeper.session.timeout
> > zookeeper.recovery.retry.intervalmill
> > hbase.rpc.timeout
> >
> > Regards,
> > Praneesh
>
>
Re: How to make the client fast fail
Posted by PRANEESH KUMAR <pr...@gmail.com>.
Hi Michael,
We can have a monitoring thread and interrupt the hbase client thread
after time out instead of doing this I want the timeout or some exception
to be thrown from the HBase client itself.
On Thu, Jun 11, 2015 at 5:16 AM, Michael Segel <mi...@hotmail.com>
wrote:
> threads?
>
> So that regardless of your hadoop settings, if you want something faster,
> you can use one thread for a timer and then the request is in another. So
> if you hit your timeout before you get a response, you can stop your thread.
> (YMMV depending on side effects… )
>
> > On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR <pr...@gmail.com>
> wrote:
> >
> > Hi,
> >
> > I have got the Connection object with default configuration, if the
> > zookeeper or HMaster or Region server is down, the client didn't fast
> fail
> > and it took almost 20 mins to thrown an error.
> >
> > What is the best configuration to make the client fast fail.
> >
> > Also what is significance of changing the following parameters.
> >
> > hbase.client.retries.number
> > zookeeper.recovery.retry
> > zookeeper.session.timeout
> > zookeeper.recovery.retry.intervalmill
> > hbase.rpc.timeout
> >
> > Regards,
> > Praneesh
>
>
RE: How to make the client fast fail
Posted by Ha...@Dell.com.
In our case (0.94.15), we had a timer to interrupt the hanging thread. Subsequently, we are able to reconnect to hbase and it all worked fine. But we observed the old zookeeper-client thread(s) still failing to connect in addition to new set of zookeeper-client thread(s) which are serving with response.
So we scored out the timer option.
Thanks,
Hari
-----Original Message-----
From: Michael Segel [mailto:michael_segel@hotmail.com]
Sent: Thursday, June 11, 2015 5:17 AM
To: user@hbase.apache.org
Subject: Re: How to make the client fast fail
threads?
So that regardless of your hadoop settings, if you want something faster, you can use one thread for a timer and then the request is in another. So if you hit your timeout before you get a response, you can stop your thread.
(YMMV depending on side effects... )
> On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR wrote:
>
> Hi,
>
> I have got the Connection object with default configuration, if the
> zookeeper or HMaster or Region server is down, the client didn't fast
> fail and it took almost 20 mins to thrown an error.
>
> What is the best configuration to make the client fast fail.
>
> Also what is significance of changing the following parameters.
>
> hbase.client.retries.number
> zookeeper.recovery.retry
> zookeeper.session.timeout
> zookeeper.recovery.retry.intervalmill
> hbase.rpc.timeout
>
> Regards,
> Praneesh
Re: How to make the client fast fail
Posted by Michael Segel <mi...@hotmail.com>.
threads?
So that regardless of your hadoop settings, if you want something faster, you can use one thread for a timer and then the request is in another. So if you hit your timeout before you get a response, you can stop your thread.
(YMMV depending on side effects… )
> On Jun 10, 2015, at 12:55 AM, PRANEESH KUMAR <pr...@gmail.com> wrote:
>
> Hi,
>
> I have got the Connection object with default configuration, if the
> zookeeper or HMaster or Region server is down, the client didn't fast fail
> and it took almost 20 mins to thrown an error.
>
> What is the best configuration to make the client fast fail.
>
> Also what is significance of changing the following parameters.
>
> hbase.client.retries.number
> zookeeper.recovery.retry
> zookeeper.session.timeout
> zookeeper.recovery.retry.intervalmill
> hbase.rpc.timeout
>
> Regards,
> Praneesh