You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Sammi Chen (Jira)" <ji...@apache.org> on 2019/12/10 11:49:00 UTC
[jira] [Updated] (HDDS-2702) Client failed to recover from ratis
AlreadyClosedException exception
[ https://issues.apache.org/jira/browse/HDDS-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sammi Chen updated HDDS-2702:
-----------------------------
Description:
Run teragen, and it failed to enter the mapreduce stage and print the following warnning message on console endlessly.
{noformat}
19/12/10 19:23:54 WARN io.KeyOutputStream: Encountered exception java.io.IOException: Unexpected Storage Container Exception: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client:client-FBD45C9313A5->RAFT is closed. on the pipeline Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes: ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172, networkLocation: /rack2, certSerialId: null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044, networkLocation: /rack2, certSerialId: null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122, networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed block length is 0, uncommitted data length is 295833 retry count 0
19/12/10 19:23:54 INFO io.BlockOutputStreamEntryPool: Allocating block with ExcludeList {datanodes = [], containerIds = [], pipelineIds = [PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
19/12/10 19:26:16 WARN io.KeyOutputStream: Encountered exception java.io.IOException: Unexpected Storage Container Exception: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client:client-7C5A7B5CFC31->RAFT is closed. on the pipeline Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes: ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172, networkLocation: /rack2, certSerialId: null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044, networkLocation: /rack2, certSerialId: null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122, networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed block length is 0, uncommitted data length is 295833 retry count 0
19/12/10 19:26:16 INFO io.BlockOutputStreamEntryPool: Allocating block with ExcludeList {datanodes = [], containerIds = [], pipelineIds = [PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
19/12/10 19:28:38 WARN io.KeyOutputStream: Encountered exception java.io.IOException: Unexpected Storage Container Exception: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client:client-B3D8C0052C4E->RAFT is closed. on the pipeline Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes: ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172, networkLocation: /rack2, certSerialId: null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044, networkLocation: /rack2, certSerialId: null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122, networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed block length is 0, uncommitted data length is 295833 retry count 0
19/12/10 19:28:38 INFO io.BlockOutputStreamEntryPool: Allocating block with ExcludeList {datanodes = [], containerIds = [], pipelineIds = [PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
{noformat}
was:
Run teragen, and it failed to enter the mapreduce stage and print the following warnning message on console endlessly.
19/12/10 19:23:54 WARN io.KeyOutputStream: Encountered exception java.io.IOException: Unexpected Storage Container Exception: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client:client-FBD45C9313A5->RAFT is closed. on the pipeline Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes: ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172, networkLocation: /rack2, certSerialId: null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044, networkLocation: /rack2, certSerialId: null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122, networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed block length is 0, uncommitted data length is 295833 retry count 0
19/12/10 19:23:54 INFO io.BlockOutputStreamEntryPool: Allocating block with ExcludeList {datanodes = [], containerIds = [], pipelineIds = [PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
19/12/10 19:26:16 WARN io.KeyOutputStream: Encountered exception java.io.IOException: Unexpected Storage Container Exception: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client:client-7C5A7B5CFC31->RAFT is closed. on the pipeline Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes: ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172, networkLocation: /rack2, certSerialId: null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044, networkLocation: /rack2, certSerialId: null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122, networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed block length is 0, uncommitted data length is 295833 retry count 0
19/12/10 19:26:16 INFO io.BlockOutputStreamEntryPool: Allocating block with ExcludeList {datanodes = [], containerIds = [], pipelineIds = [PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
19/12/10 19:28:38 WARN io.KeyOutputStream: Encountered exception java.io.IOException: Unexpected Storage Container Exception: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client:client-B3D8C0052C4E->RAFT is closed. on the pipeline Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes: ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172, networkLocation: /rack2, certSerialId: null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044, networkLocation: /rack2, certSerialId: null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122, networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed block length is 0, uncommitted data length is 295833 retry count 0
19/12/10 19:28:38 INFO io.BlockOutputStreamEntryPool: Allocating block with ExcludeList {datanodes = [], containerIds = [], pipelineIds = [PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
> Client failed to recover from ratis AlreadyClosedException exception
> --------------------------------------------------------------------
>
> Key: HDDS-2702
> URL: https://issues.apache.org/jira/browse/HDDS-2702
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Sammi Chen
> Priority: Critical
>
> Run teragen, and it failed to enter the mapreduce stage and print the following warnning message on console endlessly.
> {noformat}
> 19/12/10 19:23:54 WARN io.KeyOutputStream: Encountered exception java.io.IOException: Unexpected Storage Container Exception: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client:client-FBD45C9313A5->RAFT is closed. on the pipeline Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes: ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172, networkLocation: /rack2, certSerialId: null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044, networkLocation: /rack2, certSerialId: null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122, networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed block length is 0, uncommitted data length is 295833 retry count 0
> 19/12/10 19:23:54 INFO io.BlockOutputStreamEntryPool: Allocating block with ExcludeList {datanodes = [], containerIds = [], pipelineIds = [PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
> 19/12/10 19:26:16 WARN io.KeyOutputStream: Encountered exception java.io.IOException: Unexpected Storage Container Exception: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client:client-7C5A7B5CFC31->RAFT is closed. on the pipeline Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes: ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172, networkLocation: /rack2, certSerialId: null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044, networkLocation: /rack2, certSerialId: null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122, networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed block length is 0, uncommitted data length is 295833 retry count 0
> 19/12/10 19:26:16 INFO io.BlockOutputStreamEntryPool: Allocating block with ExcludeList {datanodes = [], containerIds = [], pipelineIds = [PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
> 19/12/10 19:28:38 WARN io.KeyOutputStream: Encountered exception java.io.IOException: Unexpected Storage Container Exception: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client:client-B3D8C0052C4E->RAFT is closed. on the pipeline Pipeline[ Id: 90deb863-e511-4a5e-ae86-dc8035e8fa7d, Nodes: ed90869c-317e-4303-8922-9fa83a3983cb{ip: 10.120.113.172, host: host172, networkLocation: /rack2, certSerialId: null}1da74a1d-f64d-4ad4-b04c-85f26687e683{ip: 10.121.124.44, host: host044, networkLocation: /rack2, certSerialId: null}515cab4b-39b5-4439-b1a8-a7b725f5784a{ip: 10.120.139.122, host: host122, networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN, leaderId:515cab4b-39b5-4439-b1a8-a7b725f5784a ]. The last committed block length is 0, uncommitted data length is 295833 retry count 0
> 19/12/10 19:28:38 INFO io.BlockOutputStreamEntryPool: Allocating block with ExcludeList {datanodes = [], containerIds = [], pipelineIds = [PipelineID=90deb863-e511-4a5e-ae86-dc8035e8fa7d]}
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org