[jira] [Commented] (HDDS-12658) Verify datanode space availability when selecting pipeline to write to

["Siddhant Sangwan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDDS-12658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17949628#comment-17949628 ] 

Siddhant Sangwan commented on HDDS-12658:
-----------------------------------------

[~peterxcli] the duplicate Jira is assigned to you - are you planning to work on this issue next? I am going through some of your other assignments in the umbrella Jira. Out of them this one seems the most important, so it's a good jira to pick up next.

If you won't be able to work on this, please let me know.

> Verify datanode space availability when selecting pipeline to write to
> ----------------------------------------------------------------------
>
>                 Key: HDDS-12658
>                 URL: https://issues.apache.org/jira/browse/HDDS-12658
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: SCM
>            Reporter: Siddhant Sangwan
>            Priority: Major
>
> When writing in Ozone, SCM selects a container to write to. We have a PipelineChoosePolicy interface that's used to select which pipeline the write will go to. The default implementation picks a pipeline randomly. However it doesn't check whether the Datanodes in this pipeline have enough disk space available. If a full pipeline is handed over to the client, it can cause writes to fail and reduce write performance.
> We do have a check later on in that code path that checks whether the chosen container has sufficient space:
> {code}
> containerInfo.getUsedBytes() + size <= this.containerSize
> {code}
> But this in itself is not sufficient.
> I propose adding some common logic to only pick a pipeline if all the Datanodes in that pipeline have sufficient available space. All implementations of PipelineChoosePolicy should use it. I need to check how this area is handling parallel writes and other synchronization/race details.
> Note that we also have another implementation of the PipelineChoosePolicy, CapacityPipelineChoosePolicy:
> {code}
> Pipeline choose policy that randomly choose pipeline with relatively lower utilization.
> {code}
> I think all policies including this one will benefit from having the proposed check.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org