You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Senthilkumar (JIRA)" <ji...@apache.org> on 2016/10/07 04:59:20 UTC
[jira] [Commented] (HDFS-10977) Balancer should query NameNode with
a timeout
[ https://issues.apache.org/jira/browse/HDFS-10977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15554136#comment-15554136 ]
Senthilkumar commented on HDFS-10977:
-------------------------------------
[~zhz] , When i was working with balancer last month i too faced this issue ,
2016-09-08 06:32:06,574 DEBUG org.apache.hadoop.ipc.Client: IPC Client (685788708) connection to host/10.103.108.
201:8020 from hadoop/host@DOMAIN sending #49230
2016-09-08 06:32:06,574 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getBlocks took 0ms
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client (685788708) connection to host/10.103.108.
201:8020 from hadoop/host@DOMAIN sending #49231
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client (685788708) connection to host/10.103.108.
201:8020 from hadoop/host@DOMAIN got value #49229
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getBlocks took 1ms
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client (685788708) connection to host/10.103.108.
201:8020 from hadoop/host@DOMAIN got value #49230
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client (685788708) connection to host/10.103.108.
201:8020 from hadoop/host@DOMAIN sending #49232
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getBlocks took 1ms
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client (685788708) connection to host/10.103.108.
201:8020 from hadoop/host@DOMAIN sending #49233
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client (685788708) connection to host/10.103.108.
201:8020 from hadoop/host@DOMAIN got value #49231
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getBlocks took 0ms
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client (685788708) connection to host/10.103.108.
201:8020 from hadoop/host@DOMAIN sending #49234
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client (685788708) connection to host/10.103.108.
201:8020 from hadoop/host@DOMAIN got value #49232
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getBlocks took 0ms
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client (685788708) connection to host/10.103.108.
201:8020 from hadoop/host@DOMAIN sending #49235
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client (685788708) connection to host/10.103.108.
201:8020 from hadoop/host@DOMAIN got value #49233
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getBlocks took 0ms
2016-09-08 06:32:06,575 DEBUG org.apache.hadoop.ipc.Client: IPC Client (685788708) connection to host/10.103.108.
201:8020 from hadoop/host@DOMAIN got value #49234
Here is the TRACE where it took few hours and no blocks found .. I ended up in Restarting balancer to make it work..
At last i started balancer with -include option ( by pulling LIVE DNs ) and it helped .. Not sure including live nodes is the right solution ?? .. If it is the case i think problem is balancer taking decomm/decommed nodes as well ??
Attaching the discussion which i started sometime back ..
> Balancer should query NameNode with a timeout
> ---------------------------------------------
>
> Key: HDFS-10977
> URL: https://issues.apache.org/jira/browse/HDFS-10977
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
> Reporter: Zhe Zhang
> Assignee: Zhe Zhang
> Attachments: HDFS-10977-reproduce.patch
>
>
> We found a case where {{Dispatcher}} was stuck at {{getBlockList}} *forever* (well, several hours when we found it).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org