You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Arul Ramachandran <ar...@gmail.com> on 2014/11/18 01:59:20 UTC
HBase concurrent.RejectedExecutionException
Hi,
Our Hbase application gets the following exception - HBase
0.96.1.2.0.6.1-101-hadoop2. I looked at the region server log and nothing
unusual is happening.
Any pointers on what else I can check? Thanks!
2014-11-16 09:49:37,686 WARN
[hbase-connection-shared-executor-pool644-t158]
AsyncProcess.sendMultiAction(AsyncProcess.java:511) - The task was rejected
by the pool. This is unexpected. Server is rs3.world.com,60020,1414690096750
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.FutureTask@f48544d rejected from
java.util.concurrent.ThreadPoolExecutor@10916cfd[Shutting down, pool size =
7, active threads = 7, queued tasks = 0, completed tasks = 932]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
at
org.apache.hadoop.hbase.client.AsyncProcess.sendMultiAction(AsyncProcess.java:506)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:461)
at
org.apache.hadoop.hbase.client.AsyncProcess.receiveMultiAction(AsyncProcess.java:700)
at
org.apache.hadoop.hbase.client.AsyncProcess.access$300(AsyncProcess.java:89)
at
org.apache.hadoop.hbase.client.AsyncProcess$1.run(AsyncProcess.java:498)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-11-16 09:49:37,686 INFO [qtp1232775351-168005]
AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:743) - : Waiting
for the global number of running tasks to be equals or less than 0,
tasksSent=3, tasksDone=2, currentTasksDone=2, tableName=group_sku_mapping
2014-11-16 09:49:37,686 INFO [qtp1232775351-168006]
AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:743) - : Waiting
for the global number of running tasks to be equals or less than 0,
tasksSent=3, tasksDone=2, currentTasksDone=2, tableName=group_sku_mapping
2014-11-16 09:49:37,686 WARN
[hbase-connection-shared-executor-pool644-t159]
AsyncProcess.sendMultiAction(AsyncProcess.java:511) - The task was rejected
by the pool. This is unexpected. Server is rs3.world.com,60020,1414690096750
Re: HBase concurrent.RejectedExecutionException
Posted by Nicolas Liochon <nk...@gmail.com>.
Hi Arul,
It's a pure client exception: it means that the client has not even tried
to send the query to the server, it failed before.
Why the client failed is another question.
I see that the pool size is 7, have you changed the default configuration?
Cheers,
Nicolas
On Tue, Nov 18, 2014 at 7:29 AM, Arul Ramachandran <ar...@gmail.com>
wrote:
> Hi Ted,
>
> I don't have the load metrics at the moment... are you suggesting this
> could be load related?
>
> Thanks
>
> On Mon, Nov 17, 2014 at 5:34 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > What was the load on rs3.world.com,60020,1414690096750 around 09:49:30 ?
> >
> > Cheers
> >
> > On Mon, Nov 17, 2014 at 4:59 PM, Arul Ramachandran <ar...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Our Hbase application gets the following exception - HBase
> > > 0.96.1.2.0.6.1-101-hadoop2. I looked at the region server log and
> nothing
> > > unusual is happening.
> > >
> > > Any pointers on what else I can check? Thanks!
> > >
> > > 2014-11-16 09:49:37,686 WARN
> > > [hbase-connection-shared-executor-pool644-t158]
> > > AsyncProcess.sendMultiAction(AsyncProcess.java:511) - The task was
> > rejected
> > > by the pool. This is unexpected. Server is rs3.world.com
> > > ,60020,1414690096750
> > >
> > > java.util.concurrent.RejectedExecutionException: Task
> > > java.util.concurrent.FutureTask@f48544d rejected from
> > > java.util.concurrent.ThreadPoolExecutor@10916cfd[Shutting down, pool
> > size
> > > =
> > > 7, active threads = 7, queued tasks = 0, completed tasks = 932]
> > >
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
> > >
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
> > >
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
> > >
> > > at
> > >
> > >
> >
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.client.AsyncProcess.sendMultiAction(AsyncProcess.java:506)
> > >
> > > at
> > >
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:461)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.client.AsyncProcess.receiveMultiAction(AsyncProcess.java:700)
> > >
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.client.AsyncProcess.access$300(AsyncProcess.java:89)
> > >
> > > at
> > >
> org.apache.hadoop.hbase.client.AsyncProcess$1.run(AsyncProcess.java:498)
> > >
> > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > >
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > >
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > >
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > >
> > > at java.lang.Thread.run(Thread.java:744)
> > >
> > > 2014-11-16 09:49:37,686 INFO [qtp1232775351-168005]
> > > AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:743) - :
> > Waiting
> > > for the global number of running tasks to be equals or less than 0,
> > > tasksSent=3, tasksDone=2, currentTasksDone=2,
> tableName=group_sku_mapping
> > >
> > > 2014-11-16 09:49:37,686 INFO [qtp1232775351-168006]
> > > AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:743) - :
> > Waiting
> > > for the global number of running tasks to be equals or less than 0,
> > > tasksSent=3, tasksDone=2, currentTasksDone=2,
> tableName=group_sku_mapping
> > >
> > > 2014-11-16 09:49:37,686 WARN
> > > [hbase-connection-shared-executor-pool644-t159]
> > > AsyncProcess.sendMultiAction(AsyncProcess.java:511) - The task was
> > rejected
> > > by the pool. This is unexpected. Server is rs3.world.com
> > > ,60020,1414690096750
> > >
> >
>
Re: HBase concurrent.RejectedExecutionException
Posted by Arul Ramachandran <ar...@gmail.com>.
Hi Ted,
I don't have the load metrics at the moment... are you suggesting this
could be load related?
Thanks
On Mon, Nov 17, 2014 at 5:34 PM, Ted Yu <yu...@gmail.com> wrote:
> What was the load on rs3.world.com,60020,1414690096750 around 09:49:30 ?
>
> Cheers
>
> On Mon, Nov 17, 2014 at 4:59 PM, Arul Ramachandran <ar...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Our Hbase application gets the following exception - HBase
> > 0.96.1.2.0.6.1-101-hadoop2. I looked at the region server log and nothing
> > unusual is happening.
> >
> > Any pointers on what else I can check? Thanks!
> >
> > 2014-11-16 09:49:37,686 WARN
> > [hbase-connection-shared-executor-pool644-t158]
> > AsyncProcess.sendMultiAction(AsyncProcess.java:511) - The task was
> rejected
> > by the pool. This is unexpected. Server is rs3.world.com
> > ,60020,1414690096750
> >
> > java.util.concurrent.RejectedExecutionException: Task
> > java.util.concurrent.FutureTask@f48544d rejected from
> > java.util.concurrent.ThreadPoolExecutor@10916cfd[Shutting down, pool
> size
> > =
> > 7, active threads = 7, queued tasks = 0, completed tasks = 932]
> >
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
> >
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
> >
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
> >
> > at
> >
> >
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
> >
> > at
> >
> >
> org.apache.hadoop.hbase.client.AsyncProcess.sendMultiAction(AsyncProcess.java:506)
> >
> > at
> > org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:461)
> >
> > at
> >
> >
> org.apache.hadoop.hbase.client.AsyncProcess.receiveMultiAction(AsyncProcess.java:700)
> >
> > at
> >
> >
> org.apache.hadoop.hbase.client.AsyncProcess.access$300(AsyncProcess.java:89)
> >
> > at
> > org.apache.hadoop.hbase.client.AsyncProcess$1.run(AsyncProcess.java:498)
> >
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >
> > at java.lang.Thread.run(Thread.java:744)
> >
> > 2014-11-16 09:49:37,686 INFO [qtp1232775351-168005]
> > AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:743) - :
> Waiting
> > for the global number of running tasks to be equals or less than 0,
> > tasksSent=3, tasksDone=2, currentTasksDone=2, tableName=group_sku_mapping
> >
> > 2014-11-16 09:49:37,686 INFO [qtp1232775351-168006]
> > AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:743) - :
> Waiting
> > for the global number of running tasks to be equals or less than 0,
> > tasksSent=3, tasksDone=2, currentTasksDone=2, tableName=group_sku_mapping
> >
> > 2014-11-16 09:49:37,686 WARN
> > [hbase-connection-shared-executor-pool644-t159]
> > AsyncProcess.sendMultiAction(AsyncProcess.java:511) - The task was
> rejected
> > by the pool. This is unexpected. Server is rs3.world.com
> > ,60020,1414690096750
> >
>
Re: HBase concurrent.RejectedExecutionException
Posted by Ted Yu <yu...@gmail.com>.
What was the load on rs3.world.com,60020,1414690096750 around 09:49:30 ?
Cheers
On Mon, Nov 17, 2014 at 4:59 PM, Arul Ramachandran <ar...@gmail.com>
wrote:
> Hi,
>
> Our Hbase application gets the following exception - HBase
> 0.96.1.2.0.6.1-101-hadoop2. I looked at the region server log and nothing
> unusual is happening.
>
> Any pointers on what else I can check? Thanks!
>
> 2014-11-16 09:49:37,686 WARN
> [hbase-connection-shared-executor-pool644-t158]
> AsyncProcess.sendMultiAction(AsyncProcess.java:511) - The task was rejected
> by the pool. This is unexpected. Server is rs3.world.com
> ,60020,1414690096750
>
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.FutureTask@f48544d rejected from
> java.util.concurrent.ThreadPoolExecutor@10916cfd[Shutting down, pool size
> =
> 7, active threads = 7, queued tasks = 0, completed tasks = 932]
>
> at
>
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
>
> at
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
>
> at
>
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
>
> at
>
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
>
> at
>
> org.apache.hadoop.hbase.client.AsyncProcess.sendMultiAction(AsyncProcess.java:506)
>
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:461)
>
> at
>
> org.apache.hadoop.hbase.client.AsyncProcess.receiveMultiAction(AsyncProcess.java:700)
>
> at
>
> org.apache.hadoop.hbase.client.AsyncProcess.access$300(AsyncProcess.java:89)
>
> at
> org.apache.hadoop.hbase.client.AsyncProcess$1.run(AsyncProcess.java:498)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:744)
>
> 2014-11-16 09:49:37,686 INFO [qtp1232775351-168005]
> AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:743) - : Waiting
> for the global number of running tasks to be equals or less than 0,
> tasksSent=3, tasksDone=2, currentTasksDone=2, tableName=group_sku_mapping
>
> 2014-11-16 09:49:37,686 INFO [qtp1232775351-168006]
> AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:743) - : Waiting
> for the global number of running tasks to be equals or less than 0,
> tasksSent=3, tasksDone=2, currentTasksDone=2, tableName=group_sku_mapping
>
> 2014-11-16 09:49:37,686 WARN
> [hbase-connection-shared-executor-pool644-t159]
> AsyncProcess.sendMultiAction(AsyncProcess.java:511) - The task was rejected
> by the pool. This is unexpected. Server is rs3.world.com
> ,60020,1414690096750
>