You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Nandakumar (Jira)" <ji...@apache.org> on 2024/09/10 06:01:00 UTC
[jira] [Commented] (HDDS-11430) [QUESTION] A bunch of irrelevant pipelines on datanode
[ https://issues.apache.org/jira/browse/HDDS-11430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880495#comment-17880495 ]
Nandakumar commented on HDDS-11430:
-----------------------------------
[~vtutrinov], we use [Github Discussion|https://github.com/apache/ozone/discussions] for questions and new idea discussions.
Can you start a new [Github Discussion|https://github.com/apache/ozone/discussions] for this?
> [QUESTION] A bunch of irrelevant pipelines on datanode
> ------------------------------------------------------
>
> Key: HDDS-11430
> URL: https://issues.apache.org/jira/browse/HDDS-11430
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.4.0
> Reporter: Vyacheslav Tutrinov
> Assignee: Nandakumar
> Priority: Critical
> Attachments: Screenshot 2024-09-04 at 19.22.13.png
>
>
> There was a following case:
> * 9 datanodes
> * a bucket with RATIS/THREE replication configuration
> * around 6Tb total disk space
> * a bunch of write operations applied and SCM started to respond with error that we haven't enough datanodes to allocate new pipeline
> * datanodes's logs contains entries about "volume space is not enough"
> * the cluster was living in that state some time (2-3 hours) and clients were continuing their attempts to write keys
> * at a certain moment the cluster was rebooted and 6 datanodes not entered to HEALTHY state
> * heap dump was fetched from one of that datanodes - a lot of SegmentedRaftLog objects was detected
> !Screenshot 2024-09-04 at 19.22.13.png|width=845,height=630!
> * these are the pipeline objects that were trying to be created earlier
> * the datanodes were not started correctly until the dfs.container.ratis.log.appender.queue.byte-limit configuration property was decreased to 100KB (the default value is 32MB)
> * without reducing the conf property value, data nodes would reach a too large heap space state
> * eventually, the datanodes removed unnecessary pipelines and entered a HEALTHY state
> h3. QUESTION
> Why did SCM try to create pipelines on full datanodes and how to avoid this?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org