You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by jeff saremi <je...@hotmail.com> on 2017/05/19 22:59:34 UTC
What is the cause for RegionTooBusyException?
We're getting errors like this. Where should we be looking into to solve this?
Failed 69261 actions: RegionTooBusyException: 12695 times, RemoteWithExtrasException: 56566 times
thanks
Jeff
Re: What is the cause for RegionTooBusyException?
Posted by Stack <st...@duboce.net>.
On Fri, May 26, 2017 at 9:47 AM, jeff saremi <je...@hotmail.com> wrote:
>
> I have an issue with this exception being thrown period. I think the
> resource management needs a lot of work. I will soon post another note
> about my impression of this whole thing.
>
>
Please. Would appreciate your input Jeff.
Thanks,
St.Ack
> Jeff
>
> ________________________________
> From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
> stack@duboce.net>
> Sent: Friday, May 26, 2017 12:05:36 AM
> To: Hbase-User
> Subject: Re: What is the cause for RegionTooBusyException?
>
> On Mon, May 22, 2017 at 9:31 AM, jeff saremi <je...@hotmail.com>
> wrote:
>
> > while I'm still trying to find anything useful in the logs, my question
> is
> > why isn't HBase self managing this?
> >
>
> It should do better here, yes (I thought TooBusy retried but I am not
> finding it at the mo.). Exception is thrown for such as the reasons James
> lists -- in essence out of resources -- including the case where we fail
> to obtain lock inside the configured timeouts (row lock on write or region
> lock doing bulk load). As James notes, you should see the too busy dumped
> into the regionserver log at time of issue. Having this, you can figure
> what resource is crimped. Is there no more detail on client side on the
> root of the TooBusy exceptions?
>
>
> Thanks,
> S
>
>
>
> >
> > ________________________________
> > From: jeff saremi <je...@hotmail.com>
> > Sent: Friday, May 19, 2017 8:18:59 PM
> > To: user@hbase.apache.org
> > Subject: Re: What is the cause for RegionTooBusyException?
> >
> > Thanks Ted. I will look deeper as you suggested
> >
> > ________________________________
> > From: Ted Yu <yu...@gmail.com>
> > Sent: Friday, May 19, 2017 4:18:12 PM
> > To: user@hbase.apache.org
> > Subject: Re: What is the cause for RegionTooBusyException?
> >
> > Have you checked region server log ?
> > Please take a look at the following method in HRegion:
> >
> > private void checkResources() throws RegionTooBusyException {
> >
> > ...
> >
> > if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {
> >
> > blockedRequestsCount.increment();
> >
> > requestFlush();
> >
> > throw new RegionTooBusyException("Above memstore limit, " +
> >
> > Which hbase release are you using ?
> >
> > Cheers
> >
> > On Fri, May 19, 2017 at 3:59 PM, jeff saremi <je...@hotmail.com>
> > wrote:
> >
> > > We're getting errors like this. Where should we be looking into to
> solve
> > > this?
> > >
> > >
> > > Failed 69261 actions: RegionTooBusyException: 12695 times,
> > > RemoteWithExtrasException: 56566 times
> > >
> > > thanks
> > >
> > > Jeff
> > >
> > >
> >
>
Re: What is the cause for RegionTooBusyException?
Posted by jeff saremi <je...@hotmail.com>.
@James, thank you very much. That was extremely helpful
________________________________
From: James Moore <jc...@hubspot.com>
Sent: Friday, May 26, 2017 10:24:42 AM
To: user@hbase.apache.org
Subject: Re: What is the cause for RegionTooBusyException?
One mechanism for revealing the error in question is to print one of the
individual exceptions which are included in the batch calls response. We
use this in a few places to allow inspection of individual Exceptions you
can see an example of how to do this over here
https://github.com/apache/hbase/blob/master/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java#L1222
On Fri, May 26, 2017 at 12:47 PM, jeff saremi <je...@hotmail.com>
wrote:
> Hi Stack
>
> no there are no details in the exception. I mentioned that in another
> thread. When you perform a Batch operation, I believe no details will be
> communicated. I am not sure about individual Put's though. That makes it
> hard to go through logs cause we don't know out of hundreds of RS's which
> logs to look at
>
> I have an issue with this exception being thrown period. I think the
> resource management needs a lot of work. I will soon post another note
> about my impression of this whole thing.
>
> Jeff
>
> ________________________________
> From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
> stack@duboce.net>
> Sent: Friday, May 26, 2017 12:05:36 AM
> To: Hbase-User
> Subject: Re: What is the cause for RegionTooBusyException?
>
> On Mon, May 22, 2017 at 9:31 AM, jeff saremi <je...@hotmail.com>
> wrote:
>
> > while I'm still trying to find anything useful in the logs, my question
> is
> > why isn't HBase self managing this?
> >
>
> It should do better here, yes (I thought TooBusy retried but I am not
> finding it at the mo.). Exception is thrown for such as the reasons James
> lists -- in essence out of resources -- including the case where we fail
> to obtain lock inside the configured timeouts (row lock on write or region
> lock doing bulk load). As James notes, you should see the too busy dumped
> into the regionserver log at time of issue. Having this, you can figure
> what resource is crimped. Is there no more detail on client side on the
> root of the TooBusy exceptions?
>
>
> Thanks,
> S
>
>
>
> >
> > ________________________________
> > From: jeff saremi <je...@hotmail.com>
> > Sent: Friday, May 19, 2017 8:18:59 PM
> > To: user@hbase.apache.org
> > Subject: Re: What is the cause for RegionTooBusyException?
> >
> > Thanks Ted. I will look deeper as you suggested
> >
> > ________________________________
> > From: Ted Yu <yu...@gmail.com>
> > Sent: Friday, May 19, 2017 4:18:12 PM
> > To: user@hbase.apache.org
> > Subject: Re: What is the cause for RegionTooBusyException?
> >
> > Have you checked region server log ?
> > Please take a look at the following method in HRegion:
> >
> > private void checkResources() throws RegionTooBusyException {
> >
> > ...
> >
> > if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {
> >
> > blockedRequestsCount.increment();
> >
> > requestFlush();
> >
> > throw new RegionTooBusyException("Above memstore limit, " +
> >
> > Which hbase release are you using ?
> >
> > Cheers
> >
> > On Fri, May 19, 2017 at 3:59 PM, jeff saremi <je...@hotmail.com>
> > wrote:
> >
> > > We're getting errors like this. Where should we be looking into to
> solve
> > > this?
> > >
> > >
> > > Failed 69261 actions: RegionTooBusyException: 12695 times,
> > > RemoteWithExtrasException: 56566 times
> > >
> > > thanks
> > >
> > > Jeff
> > >
> > >
> >
>
Re: What is the cause for RegionTooBusyException?
Posted by James Moore <jc...@hubspot.com>.
One mechanism for revealing the error in question is to print one of the
individual exceptions which are included in the batch calls response. We
use this in a few places to allow inspection of individual Exceptions you
can see an example of how to do this over here
https://github.com/apache/hbase/blob/master/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java#L1222
On Fri, May 26, 2017 at 12:47 PM, jeff saremi <je...@hotmail.com>
wrote:
> Hi Stack
>
> no there are no details in the exception. I mentioned that in another
> thread. When you perform a Batch operation, I believe no details will be
> communicated. I am not sure about individual Put's though. That makes it
> hard to go through logs cause we don't know out of hundreds of RS's which
> logs to look at
>
> I have an issue with this exception being thrown period. I think the
> resource management needs a lot of work. I will soon post another note
> about my impression of this whole thing.
>
> Jeff
>
> ________________________________
> From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <
> stack@duboce.net>
> Sent: Friday, May 26, 2017 12:05:36 AM
> To: Hbase-User
> Subject: Re: What is the cause for RegionTooBusyException?
>
> On Mon, May 22, 2017 at 9:31 AM, jeff saremi <je...@hotmail.com>
> wrote:
>
> > while I'm still trying to find anything useful in the logs, my question
> is
> > why isn't HBase self managing this?
> >
>
> It should do better here, yes (I thought TooBusy retried but I am not
> finding it at the mo.). Exception is thrown for such as the reasons James
> lists -- in essence out of resources -- including the case where we fail
> to obtain lock inside the configured timeouts (row lock on write or region
> lock doing bulk load). As James notes, you should see the too busy dumped
> into the regionserver log at time of issue. Having this, you can figure
> what resource is crimped. Is there no more detail on client side on the
> root of the TooBusy exceptions?
>
>
> Thanks,
> S
>
>
>
> >
> > ________________________________
> > From: jeff saremi <je...@hotmail.com>
> > Sent: Friday, May 19, 2017 8:18:59 PM
> > To: user@hbase.apache.org
> > Subject: Re: What is the cause for RegionTooBusyException?
> >
> > Thanks Ted. I will look deeper as you suggested
> >
> > ________________________________
> > From: Ted Yu <yu...@gmail.com>
> > Sent: Friday, May 19, 2017 4:18:12 PM
> > To: user@hbase.apache.org
> > Subject: Re: What is the cause for RegionTooBusyException?
> >
> > Have you checked region server log ?
> > Please take a look at the following method in HRegion:
> >
> > private void checkResources() throws RegionTooBusyException {
> >
> > ...
> >
> > if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {
> >
> > blockedRequestsCount.increment();
> >
> > requestFlush();
> >
> > throw new RegionTooBusyException("Above memstore limit, " +
> >
> > Which hbase release are you using ?
> >
> > Cheers
> >
> > On Fri, May 19, 2017 at 3:59 PM, jeff saremi <je...@hotmail.com>
> > wrote:
> >
> > > We're getting errors like this. Where should we be looking into to
> solve
> > > this?
> > >
> > >
> > > Failed 69261 actions: RegionTooBusyException: 12695 times,
> > > RemoteWithExtrasException: 56566 times
> > >
> > > thanks
> > >
> > > Jeff
> > >
> > >
> >
>
Re: What is the cause for RegionTooBusyException?
Posted by jeff saremi <je...@hotmail.com>.
Hi Stack
no there are no details in the exception. I mentioned that in another thread. When you perform a Batch operation, I believe no details will be communicated. I am not sure about individual Put's though. That makes it hard to go through logs cause we don't know out of hundreds of RS's which logs to look at
I have an issue with this exception being thrown period. I think the resource management needs a lot of work. I will soon post another note about my impression of this whole thing.
Jeff
________________________________
From: saint.ack@gmail.com <sa...@gmail.com> on behalf of Stack <st...@duboce.net>
Sent: Friday, May 26, 2017 12:05:36 AM
To: Hbase-User
Subject: Re: What is the cause for RegionTooBusyException?
On Mon, May 22, 2017 at 9:31 AM, jeff saremi <je...@hotmail.com> wrote:
> while I'm still trying to find anything useful in the logs, my question is
> why isn't HBase self managing this?
>
It should do better here, yes (I thought TooBusy retried but I am not
finding it at the mo.). Exception is thrown for such as the reasons James
lists -- in essence out of resources -- including the case where we fail
to obtain lock inside the configured timeouts (row lock on write or region
lock doing bulk load). As James notes, you should see the too busy dumped
into the regionserver log at time of issue. Having this, you can figure
what resource is crimped. Is there no more detail on client side on the
root of the TooBusy exceptions?
Thanks,
S
>
> ________________________________
> From: jeff saremi <je...@hotmail.com>
> Sent: Friday, May 19, 2017 8:18:59 PM
> To: user@hbase.apache.org
> Subject: Re: What is the cause for RegionTooBusyException?
>
> Thanks Ted. I will look deeper as you suggested
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> Sent: Friday, May 19, 2017 4:18:12 PM
> To: user@hbase.apache.org
> Subject: Re: What is the cause for RegionTooBusyException?
>
> Have you checked region server log ?
> Please take a look at the following method in HRegion:
>
> private void checkResources() throws RegionTooBusyException {
>
> ...
>
> if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {
>
> blockedRequestsCount.increment();
>
> requestFlush();
>
> throw new RegionTooBusyException("Above memstore limit, " +
>
> Which hbase release are you using ?
>
> Cheers
>
> On Fri, May 19, 2017 at 3:59 PM, jeff saremi <je...@hotmail.com>
> wrote:
>
> > We're getting errors like this. Where should we be looking into to solve
> > this?
> >
> >
> > Failed 69261 actions: RegionTooBusyException: 12695 times,
> > RemoteWithExtrasException: 56566 times
> >
> > thanks
> >
> > Jeff
> >
> >
>
Re: What is the cause for RegionTooBusyException?
Posted by Stack <st...@duboce.net>.
On Mon, May 22, 2017 at 9:31 AM, jeff saremi <je...@hotmail.com> wrote:
> while I'm still trying to find anything useful in the logs, my question is
> why isn't HBase self managing this?
>
It should do better here, yes (I thought TooBusy retried but I am not
finding it at the mo.). Exception is thrown for such as the reasons James
lists -- in essence out of resources -- including the case where we fail
to obtain lock inside the configured timeouts (row lock on write or region
lock doing bulk load). As James notes, you should see the too busy dumped
into the regionserver log at time of issue. Having this, you can figure
what resource is crimped. Is there no more detail on client side on the
root of the TooBusy exceptions?
Thanks,
S
>
> ________________________________
> From: jeff saremi <je...@hotmail.com>
> Sent: Friday, May 19, 2017 8:18:59 PM
> To: user@hbase.apache.org
> Subject: Re: What is the cause for RegionTooBusyException?
>
> Thanks Ted. I will look deeper as you suggested
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> Sent: Friday, May 19, 2017 4:18:12 PM
> To: user@hbase.apache.org
> Subject: Re: What is the cause for RegionTooBusyException?
>
> Have you checked region server log ?
> Please take a look at the following method in HRegion:
>
> private void checkResources() throws RegionTooBusyException {
>
> ...
>
> if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {
>
> blockedRequestsCount.increment();
>
> requestFlush();
>
> throw new RegionTooBusyException("Above memstore limit, " +
>
> Which hbase release are you using ?
>
> Cheers
>
> On Fri, May 19, 2017 at 3:59 PM, jeff saremi <je...@hotmail.com>
> wrote:
>
> > We're getting errors like this. Where should we be looking into to solve
> > this?
> >
> >
> > Failed 69261 actions: RegionTooBusyException: 12695 times,
> > RemoteWithExtrasException: 56566 times
> >
> > thanks
> >
> > Jeff
> >
> >
>
Re: What is the cause for RegionTooBusyException?
Posted by jeff saremi <je...@hotmail.com>.
while I'm still trying to find anything useful in the logs, my question is why isn't HBase self managing this?
In my 2-3 decades of using databases i had never had to stop a write operation to do anything such as compactions or whatever. The only time a write would fail would be if the db runs out of physical storage. The client could slowness due to many writes but throwing exceptions was unheard of.
________________________________
From: jeff saremi <je...@hotmail.com>
Sent: Friday, May 19, 2017 8:18:59 PM
To: user@hbase.apache.org
Subject: Re: What is the cause for RegionTooBusyException?
Thanks Ted. I will look deeper as you suggested
________________________________
From: Ted Yu <yu...@gmail.com>
Sent: Friday, May 19, 2017 4:18:12 PM
To: user@hbase.apache.org
Subject: Re: What is the cause for RegionTooBusyException?
Have you checked region server log ?
Please take a look at the following method in HRegion:
private void checkResources() throws RegionTooBusyException {
...
if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {
blockedRequestsCount.increment();
requestFlush();
throw new RegionTooBusyException("Above memstore limit, " +
Which hbase release are you using ?
Cheers
On Fri, May 19, 2017 at 3:59 PM, jeff saremi <je...@hotmail.com> wrote:
> We're getting errors like this. Where should we be looking into to solve
> this?
>
>
> Failed 69261 actions: RegionTooBusyException: 12695 times,
> RemoteWithExtrasException: 56566 times
>
> thanks
>
> Jeff
>
>
Re: What is the cause for RegionTooBusyException?
Posted by jeff saremi <je...@hotmail.com>.
Thanks Ted. I will look deeper as you suggested
________________________________
From: Ted Yu <yu...@gmail.com>
Sent: Friday, May 19, 2017 4:18:12 PM
To: user@hbase.apache.org
Subject: Re: What is the cause for RegionTooBusyException?
Have you checked region server log ?
Please take a look at the following method in HRegion:
private void checkResources() throws RegionTooBusyException {
...
if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {
blockedRequestsCount.increment();
requestFlush();
throw new RegionTooBusyException("Above memstore limit, " +
Which hbase release are you using ?
Cheers
On Fri, May 19, 2017 at 3:59 PM, jeff saremi <je...@hotmail.com> wrote:
> We're getting errors like this. Where should we be looking into to solve
> this?
>
>
> Failed 69261 actions: RegionTooBusyException: 12695 times,
> RemoteWithExtrasException: 56566 times
>
> thanks
>
> Jeff
>
>
Re: What is the cause for RegionTooBusyException?
Posted by Ted Yu <yu...@gmail.com>.
Have you checked region server log ?
Please take a look at the following method in HRegion:
private void checkResources() throws RegionTooBusyException {
...
if (this.memstoreDataSize.get() > this.blockingMemStoreSize) {
blockedRequestsCount.increment();
requestFlush();
throw new RegionTooBusyException("Above memstore limit, " +
Which hbase release are you using ?
Cheers
On Fri, May 19, 2017 at 3:59 PM, jeff saremi <je...@hotmail.com> wrote:
> We're getting errors like this. Where should we be looking into to solve
> this?
>
>
> Failed 69261 actions: RegionTooBusyException: 12695 times,
> RemoteWithExtrasException: 56566 times
>
> thanks
>
> Jeff
>
>
Re: What is the cause for RegionTooBusyException?
Posted by jeff saremi <je...@hotmail.com>.
thanks James for the hints
________________________________
From: James Moore <jc...@hubspot.com>
Sent: Friday, May 19, 2017 7:42:02 PM
To: user@hbase.apache.org
Subject: Re: What is the cause for RegionTooBusyException?
That error appears to be coming from a batch call, 12695 out of 69261
operations failed with a RegionTooBusyException Some of the causes can be.
1. A full MemStore such as if you write to the MemStore faster than it can
flush or if it's too small to fit incoming writes
2. Too many storefiles for the region without compaction. The default
number of store files which will start blocking inbound writes is 15
3. If the region is in the process of being closed but has long running
operations such as a scanner fetch active.
You should be able to see some additional info for the exact cause of the
exception in your region servers logs or in the message of a specific
RegionTooBusy exception.
cheers,
--James
On Fri, May 19, 2017 at 6:59 PM, jeff saremi <je...@hotmail.com> wrote:
> We're getting errors like this. Where should we be looking into to solve
> this?
>
>
> Failed 69261 actions: RegionTooBusyException: 12695 times,
> RemoteWithExtrasException: 56566 times
>
> thanks
>
> Jeff
>
>
Re: What is the cause for RegionTooBusyException?
Posted by James Moore <jc...@hubspot.com>.
That error appears to be coming from a batch call, 12695 out of 69261
operations failed with a RegionTooBusyException Some of the causes can be.
1. A full MemStore such as if you write to the MemStore faster than it can
flush or if it's too small to fit incoming writes
2. Too many storefiles for the region without compaction. The default
number of store files which will start blocking inbound writes is 15
3. If the region is in the process of being closed but has long running
operations such as a scanner fetch active.
You should be able to see some additional info for the exact cause of the
exception in your region servers logs or in the message of a specific
RegionTooBusy exception.
cheers,
--James
On Fri, May 19, 2017 at 6:59 PM, jeff saremi <je...@hotmail.com> wrote:
> We're getting errors like this. Where should we be looking into to solve
> this?
>
>
> Failed 69261 actions: RegionTooBusyException: 12695 times,
> RemoteWithExtrasException: 56566 times
>
> thanks
>
> Jeff
>
>