You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Shashikant Banerjee (JIRA)" <ji...@apache.org> on 2018/09/14 14:57:00 UTC

[jira] [Commented] (HDDS-461) container remains in CLOSING state in SCM forever

    [ https://issues.apache.org/jira/browse/HDDS-461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16614948#comment-16614948 ] 

Shashikant Banerjee commented on HDDS-461:
------------------------------------------

The issue seems to be happening when a closeContainer command is received at a Datanode, it gets queued up to the RaftServer and fails with NOT LEADER Exception which is ignored assuming the leader will close the container. But It may happen the datanode may die during this time ,and since the earlier exception is already ignored, SCM will never retry it on other nodes.

> container remains in CLOSING state in SCM forever
> -------------------------------------------------
>
>                 Key: HDDS-461
>                 URL: https://issues.apache.org/jira/browse/HDDS-461
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>          Components: SCM
>    Affects Versions: 0.2.1
>            Reporter: Nilotpal Nandi
>            Assignee: Shashikant Banerjee
>            Priority: Major
>         Attachments: all-node-ozone-logs-1536920345.tar.gz
>
>
> Container id # 13's state is not changing from CLOSING to CLOSED.
> {noformat}
> [root@ctr-e138-1518143905142-459606-01-000002 bin]# ./ozone scmcli info 13
> raft.rpc.type = GRPC (default)
> raft.grpc.message.size.max = 33554432 (custom)
> raft.client.rpc.retryInterval = 300 ms (default)
> raft.client.async.outstanding-requests.max = 100 (default)
> raft.client.async.scheduler-threads = 3 (default)
> raft.grpc.flow.control.window = 1MB (=1048576) (default)
> raft.grpc.message.size.max = 33554432 (custom)
> raft.client.rpc.request.timeout = 3000 ms (default)
> Container id: 13
> Container State: OPEN
> Container Path: /tmp/hadoop-root/dfs/data/hdds/de0a9e01-4a12-40e3-b567-51b9bd83248e/current/containerDir0/13/metadata
> Container Metadata:
> LeaderID: ctr-e138-1518143905142-459606-01-000003.hwx.site
> Datanodes: [ctr-e138-1518143905142-459606-01-000007.hwx.site,ctr-e138-1518143905142-459606-01-000008.hwx.site,ctr-e138-1518143905142-459606-01-000003.hwx.site]{noformat}
>  
> snippet of scmcli list :
> {noformat}
> {
>  "state" : "CLOSING",
>  "replicationFactor" : "THREE",
>  "replicationType" : "RATIS",
>  "allocatedBytes" : 4831838208,
>  "usedBytes" : 4831838208,
>  "numberOfKeys" : 0,
>  "lastUsed" : 4391827471,
>  "stateEnterTime" : 5435591457,
>  "owner" : "f8332db1-b8b1-4077-a9ea-097033d074b7",
>  "containerID" : 13,
>  "deleteTransactionId" : 0,
>  "containerOpen" : true
> }{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org