You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Michael Stack (Jira)" <ji...@apache.org> on 2019/11/04 21:40:00 UTC
[jira] [Comment Edited] (HDFS-13613) RegionServer log is flooded
with "Execution rejected, Executing in current thread"
[ https://issues.apache.org/jira/browse/HDFS-13613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16967021#comment-16967021 ]
Michael Stack edited comment on HDFS-13613 at 11/4/19 9:39 PM:
---------------------------------------------------------------
Just ran into this one. Thread dump showded loads of threads BLOCKED here:
{code}
"RpcServer.default.FPBQ.Fifo.handler=85,queue=25,port=16020" #137 daemon prio=5 os_prio=0 cpu=85786.24ms elapsed=157927.35s tid=0x00007f3dddad6000 nid=0xf390 waiting for monitor entry [0x00007f3dd21a9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.log4j.Category.callAppenders(Category.java:204)
- waiting to lock <0x000000008080c258> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:305)
at org.apache.hadoop.hdfs.DFSClient$2.rejectedExecution(DFSClient.java:2904)
{code}
i.e. trying to log the above useless message.
I turned off logging but we still go into the logging system and hit the BLOCKED section.
The RS backs up, fills all call queues. Nothing can come in the front door. We start to burn all CPUs.
For context, running heavy load. HDFS slows down. HBase spews complaint that syncs are costing > 100ms. Then this phenomenon takes off....
When load lightens, we seem to be get past the torrent but while it is going on, the backed up RS is not allowing access to any of its hosted data.
was (Author: stack):
Just ran into this one. Thread dump showded loads of threads BLOCKED here:
{code}
"RpcServer.default.FPBQ.Fifo.handler=85,queue=25,port=16020" #137 daemon prio=5 os_prio=0 cpu=85786.24ms elapsed=157927.35s tid=0x00007f3dddad6000 nid=0xf390 waiting for monitor entry [0x00007f3dd21a9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.log4j.Category.callAppenders(Category.java:204)
- waiting to lock <0x000000008080c258> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:305)
at org.apache.hadoop.hdfs.DFSClient$2.rejectedExecution(DFSClient.java:2904)
{code}
i.e. trying to log the above useless message.
I turned off logging but we still go into the logging system and hit the BLOCKED section.
The RS backs up, fills all call queues. Nothing can come in the front door. We start to burn all CPUs.
> RegionServer log is flooded with "Execution rejected, Executing in current thread"
> ----------------------------------------------------------------------------------
>
> Key: HDFS-13613
> URL: https://issues.apache.org/jira/browse/HDFS-13613
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.0
> Environment: CDH 5.13, HBase RegionServer, Kerberized, hedged read
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> In the log of a HBase RegionServer with hedged read, we saw the following message flooding the log file.
> {noformat}
> 2018-05-19 17:22:55,691 INFO org.apache.hadoop.hdfs.DFSClient: Execution rejected, Executing in current thread
> 2018-05-19 17:22:55,692 INFO org.apache.hadoop.hdfs.DFSClient: Execution rejected, Executing in current thread
> 2018-05-19 17:22:55,695 INFO org.apache.hadoop.hdfs.DFSClient: Execution rejected, Executing in current thread
> 2018-05-19 17:22:55,696 INFO org.apache.hadoop.hdfs.DFSClient: Execution rejected, Executing in current thread
> 2018-05-19 17:22:55,696 INFO org.apache.hadoop.hdfs.DFSClient: Execution rejected, Executing in current thread
> ....
> {noformat}
> Sometimes the RS spits tens of thousands of lines of this message in a minute. We should do something to stop this message flooding the log file. Also, we should make this message more actionable. Discussed with [~huaxiang], this message can appear if there are stale DataNodes.
> I believe this issue existed since HDFS-5776.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org