You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "jojochuang (via GitHub)" <gi...@apache.org> on 2026/01/03 14:21:28 UTC
[PR] HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated [ozone]
jojochuang opened a new pull request, #9580:
URL: https://github.com/apache/ozone/pull/9580
## What changes were proposed in this pull request?
HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated
Please describe your PR in detail:
* Similar to HDDS-14199 but for Ratis pipelines.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13579
## How was this patch tested?
Doc only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9580:
URL: https://github.com/apache/ozone/pull/9580#issuecomment-3905639213
Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9580:
URL: https://github.com/apache/ozone/pull/9580#issuecomment-3865801817
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated [ozone]
Posted by "ashishkumar50 (via GitHub)" <gi...@apache.org>.
ashishkumar50 commented on code in PR #9580:
URL: https://github.com/apache/ozone/pull/9580#discussion_r2660630869
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.
+
+1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+ * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
+ across the entire cluster. This acts as a final cap on the total number of pipelines.
+ * **Default Value**: `0` (which means no global limit is enforced by default).
+
+2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
+ * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+ every datanode. This is one of two ways to calculate a cluster-wide target.
+ * **Default Value**: `2`
+ * **Calculation**: If this is set, the target is `(<this value> * <number of healthy datanodes>) / 3`.
+
+3. **Datanode-level Dynamic Limit (`ozone.scm.pipeline.per.metadata.disk`)**
+ * **Description**: This property is used only when `ozone.scm.datanode.pipeline.limit` is explicitly set to `0`.
+ It calculates a dynamic limit for each datanode based on its available metadata disks.
+ * **Default Value**: `2`
+ * **Calculation**: The limit for each datanode is
+ `(<this value> * <number of metadata disks on that datanode>)`.
+ The total cluster-wide target is the sum of all individual datanode limits, divided by 3.
+
+#### How Limits are Applied
+
+SCM first calculates a target number of pipelines based on either the **Datanode-level Fixed Limit** or the
+**Datanode-level Dynamic Limit**. It then compares this calculated target to the **Cluster-wide Limit**. The
+**lowest value** is used as the final target for the number of open pipelines.
+
+**Example (Dynamic Limit):**
+
+Consider a cluster with **10 healthy datanodes**.
+* **8 datanodes** have 4 metadata disks each.
+* **2 datanodes** have 2 metadata disks each.
+
+And the configuration is:
+* `ozone.scm.ratis.pipeline.limit` = **30** (A global cap is set)
+* `ozone.scm.datanode.pipeline.limit` = **0** (Use dynamic calculation)
+* `ozone.scm.pipeline.per.metadata.disk` = **2** (Default)
+
+**Calculation Steps:**
+1. Calculate the limit for the first group of datanodes: `8 datanodes * (2 pipelines/disk * 4 disks/datanode) = 64 pipelines`
+2. Calculate the limit for the second group of datanodes: `2 datanodes * (2 pipelines/disk * 2 disks/datanode) = 8 pipelines`
+3. Calculate the total raw target from the dynamic limit: `(64 + 8) / 3 = 24`
+4. Compare with the global limit: `min(24, 30) = 24`
+
+SCM will attempt to create and maintain approximately **24** open, FACTOR_THREE Ratis pipelines.
+
+**Production Recommendation:**
+
+For most production deployments, using the dynamic per-disk limit (`ozone.scm.datanode.pipeline.limit=0`) is
+recommended, as it allows the cluster to scale pipeline capacity naturally with its resources. You can use the
+global limit (`ozone.scm.ratis.pipeline.limit`) as a safety cap if needed. A good starting value for
+`ozone.scm.pipeline.per.metadata.disk` is **2**. Monitor the `NumOpenPipelines` metric in SCM to see if the
+actual number of pipelines aligns with your configured targets.
Review Comment:
I think `NumOpenPipelines` metrics doesn't exist. We can either use admin command to see number of open pipelines or may use recon as well to see the open pipelines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #9580:
URL: https://github.com/apache/ozone/pull/9580#discussion_r2669134768
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
Review Comment:
> ... FACTOR_THREE ...
Let's use [ReplicationFactor.THREE](https://github.com/apache/ozone/blob/fd181c69464afd3ed3240b376b81f7b81f0806e9/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/ReplicationFactor.java#L27), which is a client API.
> ... three properties ... that define the maximum
three configuration properties ... that limit the
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #9580:
URL: https://github.com/apache/ozone/pull/9580#issuecomment-3762431931
Ok. updated based on Nicholas' comment. I am going to go over this again since it's been awhile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #9580:
URL: https://github.com/apache/ozone/pull/9580#discussion_r2693610074
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.
+
+1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+ * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
+ across the entire cluster. This acts as a final cap on the total number of pipelines.
+ * **Default Value**: `0` (which means no global limit is enforced by default).
+
+2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
+ * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+ every datanode. This is one of two ways to calculate a cluster-wide target.
+ * **Default Value**: `2`
+ * **Calculation**: If this is set, the target is `(<this value> * <number of healthy datanodes>) / 3`.
+
+3. **Datanode-level Dynamic Limit (`ozone.scm.pipeline.per.metadata.disk`)**
+ * **Description**: This property is used only when `ozone.scm.datanode.pipeline.limit` is explicitly set to `0`.
+ It calculates a dynamic limit for each datanode based on its available metadata disks.
+ * **Default Value**: `2`
+ * **Calculation**: The limit for each datanode is
+ `(<this value> * <number of metadata disks on that datanode>)`.
+ The total cluster-wide target is the sum of all individual datanode limits, divided by 3.
+
+#### How Limits are Applied
+
+SCM first calculates a target number of pipelines based on either the **Datanode-level Fixed Limit** or the
+**Datanode-level Dynamic Limit**. It then compares this calculated target to the **Cluster-wide Limit**. The
+**lowest value** is used as the final target for the number of open pipelines.
+
+**Example (Dynamic Limit):**
+
+Consider a cluster with **10 healthy datanodes**.
+* **8 datanodes** have 4 metadata disks each.
+* **2 datanodes** have 2 metadata disks each.
+
+And the configuration is:
+* `ozone.scm.ratis.pipeline.limit` = **30** (A global cap is set)
+* `ozone.scm.datanode.pipeline.limit` = **0** (Use dynamic calculation)
+* `ozone.scm.pipeline.per.metadata.disk` = **2** (Default)
+
+**Calculation Steps:**
+1. Calculate the limit for the first group of datanodes: `8 datanodes * (2 pipelines/disk * 4 disks/datanode) = 64 pipelines`
+2. Calculate the limit for the second group of datanodes: `2 datanodes * (2 pipelines/disk * 2 disks/datanode) = 8 pipelines`
+3. Calculate the total raw target from the dynamic limit: `(64 + 8) / 3 = 24`
+4. Compare with the global limit: `min(24, 30) = 24`
+
+SCM will attempt to create and maintain approximately **24** open, FACTOR_THREE Ratis pipelines.
+
+**Production Recommendation:**
+
+For most production deployments, using the dynamic per-disk limit (`ozone.scm.datanode.pipeline.limit=0`) is
+recommended, as it allows the cluster to scale pipeline capacity naturally with its resources. You can use the
+global limit (`ozone.scm.ratis.pipeline.limit`) as a safety cap if needed. A good starting value for
+`ozone.scm.pipeline.per.metadata.disk` is **2**. Monitor the section **Pipeline Statistics** in SCM web UI, or run
+the command `ozone admin pipeline list` to see if the actual number of pipelines aligns with your configured targets.
Review Comment:
There should a tradeoff of having a lot of concurrent pipelines. This might be worth documenting.
From my understanding, the higher number of pipelines, there will be higher number concurrent storage containers. If one DN is down and all the pipelines are closed, we might end up with a lot of small containers, which might have overhead long term.
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.
+
+1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+ * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
+ across the entire cluster. This acts as a final cap on the total number of pipelines.
+ * **Default Value**: `0` (which means no global limit is enforced by default).
+
+2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
+ * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+ every datanode. This is one of two ways to calculate a cluster-wide target.
+ * **Default Value**: `2`
+ * **Calculation**: If this is set, the target is `(<this value> * <number of healthy datanodes>) / 3`.
+
+3. **Datanode-level Dynamic Limit (`ozone.scm.pipeline.per.metadata.disk`)**
+ * **Description**: This property is used only when `ozone.scm.datanode.pipeline.limit` is explicitly set to `0`.
+ It calculates a dynamic limit for each datanode based on its available metadata disks.
+ * **Default Value**: `2`
+ * **Calculation**: The limit for each datanode is
+ `(<this value> * <number of metadata disks on that datanode>)`.
+ The total cluster-wide target is the sum of all individual datanode limits, divided by 3.
+
+#### How Limits are Applied
+
+SCM first calculates a target number of pipelines based on either the **Datanode-level Fixed Limit** or the
+**Datanode-level Dynamic Limit**. It then compares this calculated target to the **Cluster-wide Limit**. The
+**lowest value** is used as the final target for the number of open pipelines.
+
+**Example (Dynamic Limit):**
+
+Consider a cluster with **10 healthy datanodes**.
+* **8 datanodes** have 4 metadata disks each.
+* **2 datanodes** have 2 metadata disks each.
+
+And the configuration is:
+* `ozone.scm.ratis.pipeline.limit` = **30** (A global cap is set)
+* `ozone.scm.datanode.pipeline.limit` = **0** (Use dynamic calculation)
+* `ozone.scm.pipeline.per.metadata.disk` = **2** (Default)
+
+**Calculation Steps:**
+1. Calculate the limit for the first group of datanodes: `8 datanodes * (2 pipelines/disk * 4 disks/datanode) = 64 pipelines`
+2. Calculate the limit for the second group of datanodes: `2 datanodes * (2 pipelines/disk * 2 disks/datanode) = 8 pipelines`
+3. Calculate the total raw target from the dynamic limit: `(64 + 8) / 3 = 24`
+4. Compare with the global limit: `min(24, 30) = 24`
+
+SCM will attempt to create and maintain approximately **24** open, FACTOR_THREE Ratis pipelines.
+
+**Production Recommendation:**
+
+For most production deployments, using the dynamic per-disk limit (`ozone.scm.datanode.pipeline.limit=0`) is
+recommended, as it allows the cluster to scale pipeline capacity naturally with its resources. You can use the
+global limit (`ozone.scm.ratis.pipeline.limit`) as a safety cap if needed. A good starting value for
+`ozone.scm.pipeline.per.metadata.disk` is **2**. Monitor the section **Pipeline Statistics** in SCM web UI, or run
+the command `ozone admin pipeline list` to see if the actual number of pipelines aligns with your configured targets.
Review Comment:
I think there should a tradeoff of having a lot of concurrent pipelines. This might be worth documenting.
From my understanding, the higher number of pipelines, there will be higher number concurrent storage containers. If one DN is down and all the pipelines are closed, we might end up with a lot of small containers, which might have overhead long term.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #9580: HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated
URL: https://github.com/apache/ozone/pull/9580
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on PR #9580:
URL: https://github.com/apache/ozone/pull/9580#issuecomment-3707358220
@jojochuang , thanks for working on this!
Could you limit the line length to 120 characters like the code? It is easier to comment on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #9580:
URL: https://github.com/apache/ozone/pull/9580#issuecomment-3707491902
thanks for review. updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #9580:
URL: https://github.com/apache/ozone/pull/9580#discussion_r2666366553
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.
+
+1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+ * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
+ across the entire cluster. This acts as a final cap on the total number of pipelines.
+ * **Default Value**: `0` (which means no global limit is enforced by default).
+
+2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
+ * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+ every datanode. This is one of two ways to calculate a cluster-wide target.
+ * **Default Value**: `2`
+ * **Calculation**: If this is set, the target is `(<this value> * <number of healthy datanodes>) / 3`.
+
+3. **Datanode-level Dynamic Limit (`ozone.scm.pipeline.per.metadata.disk`)**
+ * **Description**: This property is used only when `ozone.scm.datanode.pipeline.limit` is explicitly set to `0`.
+ It calculates a dynamic limit for each datanode based on its available metadata disks.
+ * **Default Value**: `2`
+ * **Calculation**: The limit for each datanode is
+ `(<this value> * <number of metadata disks on that datanode>)`.
+ The total cluster-wide target is the sum of all individual datanode limits, divided by 3.
+
+#### How Limits are Applied
+
+SCM first calculates a target number of pipelines based on either the **Datanode-level Fixed Limit** or the
+**Datanode-level Dynamic Limit**. It then compares this calculated target to the **Cluster-wide Limit**. The
+**lowest value** is used as the final target for the number of open pipelines.
+
+**Example (Dynamic Limit):**
+
+Consider a cluster with **10 healthy datanodes**.
+* **8 datanodes** have 4 metadata disks each.
+* **2 datanodes** have 2 metadata disks each.
+
+And the configuration is:
+* `ozone.scm.ratis.pipeline.limit` = **30** (A global cap is set)
+* `ozone.scm.datanode.pipeline.limit` = **0** (Use dynamic calculation)
+* `ozone.scm.pipeline.per.metadata.disk` = **2** (Default)
+
+**Calculation Steps:**
+1. Calculate the limit for the first group of datanodes: `8 datanodes * (2 pipelines/disk * 4 disks/datanode) = 64 pipelines`
+2. Calculate the limit for the second group of datanodes: `2 datanodes * (2 pipelines/disk * 2 disks/datanode) = 8 pipelines`
+3. Calculate the total raw target from the dynamic limit: `(64 + 8) / 3 = 24`
+4. Compare with the global limit: `min(24, 30) = 24`
+
+SCM will attempt to create and maintain approximately **24** open, FACTOR_THREE Ratis pipelines.
+
+**Production Recommendation:**
+
+For most production deployments, using the dynamic per-disk limit (`ozone.scm.datanode.pipeline.limit=0`) is
+recommended, as it allows the cluster to scale pipeline capacity naturally with its resources. You can use the
+global limit (`ozone.scm.ratis.pipeline.limit`) as a safety cap if needed. A good starting value for
+`ozone.scm.pipeline.per.metadata.disk` is **2**. Monitor the `NumOpenPipelines` metric in SCM to see if the
+actual number of pipelines aligns with your configured targets.
Review Comment:
To clarify, the SCM web UI has a section **Pipeline Statistics**, and it pulls the metrics from:
```
{
"name": "Hadoop:service=SCMPipelineManager,name=SCMPipelineManagerInfo",
"modelerType": "org.apache.hadoop.hdds.scm.pipeline.PipelineManagerImpl",
"PipelineInfo": [
{
"key": "CLOSED",
"value": 0
},
{
"key": "ALLOCATED",
"value": 0
},
{
"key": "OPEN",
"value": 1
},
{
"key": "DORMANT",
"value": 0
}
]
},
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #9580:
URL: https://github.com/apache/ozone/pull/9580#discussion_r2669180459
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.
+
+1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+ * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
+ across the entire cluster. This acts as a final cap on the total number of pipelines.
+ * **Default Value**: `0` (which means no global limit is enforced by default).
Review Comment:
> ... no global limit is enforced by default).
no global limit by default).
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.
+
+1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+ * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
+ across the entire cluster. This acts as a final cap on the total number of pipelines.
+ * **Default Value**: `0` (which means no global limit is enforced by default).
+
+2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
+ * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+ every datanode. This is one of two ways to calculate a cluster-wide target.
+ * **Default Value**: `2`
Review Comment:
This is tricky since we have a bug; see HDDS-14369.
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
Review Comment:
> ... FACTOR_THREE ...
Let's use [ReplicationFactor.THREE](https://github.com/apache/ozone/blob/fd181c69464afd3ed3240b376b81f7b81f0806e9/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/ReplicationFactor.java#L27), which is a client API.
> ... three properties ...
three configuration properties
> ... that define the maximum
that limit the
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
Review Comment:
> ... a cluster-wide level, datanode level, and metadata disk level, respectively.
a cluster-wide level and a datanode level.
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.
Review Comment:
Replace this sentence with
- "The number of pipelines created by SCM is restricted by these limits."
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.
+
+1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+ * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
+ across the entire cluster. This acts as a final cap on the total number of pipelines.
+ * **Default Value**: `0` (which means no global limit is enforced by default).
+
+2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
+ * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+ every datanode. This is one of two ways to calculate a cluster-wide target.
+ * **Default Value**: `2`
+ * **Calculation**: If this is set, the target is `(<this value> * <number of healthy datanodes>) / 3`.
Review Comment:
```md
* **Cluster-wide Limit Calculation**: If this is set, the cluster-wide limit is
`(<this value> * <number of healthy datanodes>) / 3`.
```
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.
+
+1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+ * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
+ across the entire cluster. This acts as a final cap on the total number of pipelines.
+ * **Default Value**: `0` (which means no global limit is enforced by default).
+
+2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
+ * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+ every datanode. This is one of two ways to calculate a cluster-wide target.
+ * **Default Value**: `2`
+ * **Calculation**: If this is set, the target is `(<this value> * <number of healthy datanodes>) / 3`.
+
+3. **Datanode-level Dynamic Limit (`ozone.scm.pipeline.per.metadata.disk`)**
+ * **Description**: This property is used only when `ozone.scm.datanode.pipeline.limit` is explicitly set to `0`.
Review Comment:
> ... This property is used ... is explicitly set to `0`.
This property takes effect ... is not set to a positive number.
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.
+
+1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+ * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
+ across the entire cluster. This acts as a final cap on the total number of pipelines.
+ * **Default Value**: `0` (which means no global limit is enforced by default).
+
+2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
+ * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+ every datanode. This is one of two ways to calculate a cluster-wide target.
+ * **Default Value**: `2`
+ * **Calculation**: If this is set, the target is `(<this value> * <number of healthy datanodes>) / 3`.
+
+3. **Datanode-level Dynamic Limit (`ozone.scm.pipeline.per.metadata.disk`)**
+ * **Description**: This property is used only when `ozone.scm.datanode.pipeline.limit` is explicitly set to `0`.
+ It calculates a dynamic limit for each datanode based on its available metadata disks.
+ * **Default Value**: `2`
+ * **Calculation**: The limit for each datanode is
+ `(<this value> * <number of metadata disks on that datanode>)`.
+ The total cluster-wide target is the sum of all individual datanode limits, divided by 3.
Review Comment:
This seems not true (or I cannot find the code enforcing it). I do see that the value is used for filtering datanodes.
https://github.com/apache/ozone/blob/4fd83602c3080a8174876bc9c19c356af857fd56/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java#L104-L131
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.
+
+1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+ * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
+ across the entire cluster. This acts as a final cap on the total number of pipelines.
+ * **Default Value**: `0` (which means no global limit is enforced by default).
+
+2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
+ * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+ every datanode. This is one of two ways to calculate a cluster-wide target.
Review Comment:
Remove the sentence "This is one of two ways to calculate a cluster-wide target." since the Dynamic Limit
is not used in RatisPipelineProvider.exceedPipelineNumberLimit(..).
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.
+
+1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+ * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
Review Comment:
Remove FACTOR_THREE since the code actually includes ReplicationFactor.ONE pipeline; see https://github.com/apache/ozone/blob/4fd83602c3080a8174876bc9c19c356af857fd56/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java#L123-L128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #9580:
URL: https://github.com/apache/ozone/pull/9580#discussion_r2669448953
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.
+
+1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+ * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
+ across the entire cluster. This acts as a final cap on the total number of pipelines.
+ * **Default Value**: `0` (which means no global limit is enforced by default).
+
+2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
+ * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+ every datanode. This is one of two ways to calculate a cluster-wide target.
+ * **Default Value**: `2`
+ * **Calculation**: If this is set, the target is `(<this value> * <number of healthy datanodes>) / 3`.
+
+3. **Datanode-level Dynamic Limit (`ozone.scm.pipeline.per.metadata.disk`)**
+ * **Description**: This property is used only when `ozone.scm.datanode.pipeline.limit` is explicitly set to `0`.
+ It calculates a dynamic limit for each datanode based on its available metadata disks.
+ * **Default Value**: `2`
+ * **Calculation**: The limit for each datanode is
+ `(<this value> * <number of metadata disks on that datanode>)`.
+ The total cluster-wide target is the sum of all individual datanode limits, divided by 3.
Review Comment:
Remove this section since it seems not true (or I cannot find the code enforcing it). I do see that the value is used for filtering datanodes.
https://github.com/apache/ozone/blob/4fd83602c3080a8174876bc9c19c356af857fd56/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java#L104-L131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #9580:
URL: https://github.com/apache/ozone/pull/9580#discussion_r2693610074
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.
+
+1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+ * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
+ across the entire cluster. This acts as a final cap on the total number of pipelines.
+ * **Default Value**: `0` (which means no global limit is enforced by default).
+
+2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
+ * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+ every datanode. This is one of two ways to calculate a cluster-wide target.
+ * **Default Value**: `2`
+ * **Calculation**: If this is set, the target is `(<this value> * <number of healthy datanodes>) / 3`.
+
+3. **Datanode-level Dynamic Limit (`ozone.scm.pipeline.per.metadata.disk`)**
+ * **Description**: This property is used only when `ozone.scm.datanode.pipeline.limit` is explicitly set to `0`.
+ It calculates a dynamic limit for each datanode based on its available metadata disks.
+ * **Default Value**: `2`
+ * **Calculation**: The limit for each datanode is
+ `(<this value> * <number of metadata disks on that datanode>)`.
+ The total cluster-wide target is the sum of all individual datanode limits, divided by 3.
+
+#### How Limits are Applied
+
+SCM first calculates a target number of pipelines based on either the **Datanode-level Fixed Limit** or the
+**Datanode-level Dynamic Limit**. It then compares this calculated target to the **Cluster-wide Limit**. The
+**lowest value** is used as the final target for the number of open pipelines.
+
+**Example (Dynamic Limit):**
+
+Consider a cluster with **10 healthy datanodes**.
+* **8 datanodes** have 4 metadata disks each.
+* **2 datanodes** have 2 metadata disks each.
+
+And the configuration is:
+* `ozone.scm.ratis.pipeline.limit` = **30** (A global cap is set)
+* `ozone.scm.datanode.pipeline.limit` = **0** (Use dynamic calculation)
+* `ozone.scm.pipeline.per.metadata.disk` = **2** (Default)
+
+**Calculation Steps:**
+1. Calculate the limit for the first group of datanodes: `8 datanodes * (2 pipelines/disk * 4 disks/datanode) = 64 pipelines`
+2. Calculate the limit for the second group of datanodes: `2 datanodes * (2 pipelines/disk * 2 disks/datanode) = 8 pipelines`
+3. Calculate the total raw target from the dynamic limit: `(64 + 8) / 3 = 24`
+4. Compare with the global limit: `min(24, 30) = 24`
+
+SCM will attempt to create and maintain approximately **24** open, FACTOR_THREE Ratis pipelines.
+
+**Production Recommendation:**
+
+For most production deployments, using the dynamic per-disk limit (`ozone.scm.datanode.pipeline.limit=0`) is
+recommended, as it allows the cluster to scale pipeline capacity naturally with its resources. You can use the
+global limit (`ozone.scm.ratis.pipeline.limit`) as a safety cap if needed. A good starting value for
+`ozone.scm.pipeline.per.metadata.disk` is **2**. Monitor the section **Pipeline Statistics** in SCM web UI, or run
+the command `ozone admin pipeline list` to see if the actual number of pipelines aligns with your configured targets.
Review Comment:
I think there should a tradeoff of having a lot of concurrent pipelines. This might be worth documenting.
Here are a few I can thinks of
- Each Ratis group takes some resources (e.g. 8MB for the write buffer IIRC)
- Larger number of pipelines increase the load on the metadata volume which might cause contention
- The higher number of pipelines, there will be higher number concurrent storage containers. If one DN is down and all the pipelines are closed, we might end up with a lot of small containers, which might have overhead long term.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated [ozone]
Posted by "szetszwo (via GitHub)" <gi...@apache.org>.
szetszwo commented on code in PR #9580:
URL: https://github.com/apache/ozone/pull/9580#discussion_r2669395340
##########
hadoop-hdds/docs/content/feature/multi-raft-support.md:
##########
@@ -69,6 +69,64 @@ Ratis handles concurrent logs per node.
This property is effective only when the previous property is set to 0.
The value of this property must be greater than 0.
+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.
+
+1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+ * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
+ across the entire cluster. This acts as a final cap on the total number of pipelines.
+ * **Default Value**: `0` (which means no global limit is enforced by default).
+
+2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
+ * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+ every datanode. This is one of two ways to calculate a cluster-wide target.
+ * **Default Value**: `2`
+ * **Calculation**: If this is set, the target is `(<this value> * <number of healthy datanodes>) / 3`.
Review Comment:
```md
* **Cluster-wide Limit Calculation**: If this property is set,
the number of pipelines in the cluster is in addition limited by
`(<this value> * <number of healthy datanodes>) / 3`.
```
(revised)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org