You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "YutaLin (via GitHub)" <gi...@apache.org> on 2026/01/08 04:51:34 UTC
[PR] HDDS-14286. [Website v2] [Docs] [Core Concepts] Write pipelines [ozone-site]
YutaLin opened a new pull request, #224:
URL: https://github.com/apache/ozone-site/pull/224
## What changes were proposed in this pull request?
Complete the page https://ozone-site-v2.staged.apache.org/docs/core-concepts/replication/write-pipelines
using the content from this page in the open PR https://github.com/apache/ozone-site/pull/149/changes#diff-5829e36dd50f890ffd26a6b91f3cf4e847fdda51f0374e2bb38d14e343651809
## What is the link to the Apache Jira?
https://issues.apache.org/jira/browse/HDDS-14286
## How was this patch tested?
Check off which of the following tests were done on this change. If additional testing was done, please elaborate here as well.
- [x] The CI checks on my fork are passing (https://github.com/YutaLin/ozone-site/actions/runs/20805617805)
- [x] I verified the rendered content using a local preview
- [x] I manually verified the steps provided in this change work as described
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14286. [Website v2] [Docs] [Core Concepts] Write pipelines [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang merged PR #224:
URL: https://github.com/apache/ozone-site/pull/224
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14286. [Website v2] [Docs] [Core Concepts] Write pipelines [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #224:
URL: https://github.com/apache/ozone-site/pull/224#issuecomment-3742329699
Merged. Thanks @YutaLin @peterxcli
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14286. [Website v2] [Docs] [Core Concepts] Write pipelines [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #224:
URL: https://github.com/apache/ozone-site/pull/224#discussion_r2675087373
##########
docs/03-core-concepts/02-replication/02-write-pipelines.md:
##########
@@ -1,5 +1,298 @@
+---
+sidebar_label: Write Pipelines
+---
+
# Write Pipelines
-**TODO:** File a subtask under [HDDS-9857](https://issues.apache.org/jira/browse/HDDS-9857) and complete this page or section.
+Write pipelines are a fundamental component of Apache Ozone's storage architecture, enabling reliable data storage across distributed nodes. This document provides a comprehensive overview of write pipelines, covering both replication and erasure coding approaches, their architecture, implementation details, and usage patterns.
+
+## What are Write Pipelines?
+
+Write pipelines are groups of Datanodes that work together as a unit to store and replicate data in Ozone. They serve as the foundation for Ozone's data redundancy strategy, providing:
+
+- A coordinated path for write operations across multiple nodes
+- Consistency guarantees for data replication
+- Efficient management of data distribution and storage
+
+The Storage Container Manager (SCM) is responsible for creating and managing write pipelines, selecting appropriate Datanodes based on factors like availability, capacity, and network topology.
+
+## Pipeline Types
+
+Ozone supports different types of write pipelines to accommodate various durability and storage efficiency requirements:
+
+### 1. Ratis Pipelines (Replicated)
+
+Ratis pipelines use the [Apache Ratis](https://ratis.apache.org/) implementation of the Raft consensus protocol for strongly consistent replication.
+
+- **Structure**: Typically consists of three Datanodes (one leader, multiple followers)
+- **Consistency**: Provides strong consistency through synchronous replication
+- **Durability**: Data is fully replicated on all nodes in the pipeline
+- **Use Case**: Default replication strategy for most Ozone deployments
+
+
+
+#### Ratis Pipeline V1: Async API
+
+The original Ozone replication pipeline (V1) uses the Ratis Async API for data replication across multiple Datanodes:
+
+1. Client buffers data locally until a certain threshold is reached
+2. Data is sent to the leader Datanode in the pipeline
+3. Leader replicates data to follower Datanodes
+4. Once a quorum of Datanodes acknowledge the write, the operation is considered successful
+
+This approach ensures data consistency but has some limitations in terms of network topology awareness and buffer handling efficiency.
+
+#### Ratis Pipeline V2: Streaming API
Review Comment:
on a second thought, we don't have to, since the Apache Ozone doc has Ratis streaming content already: https://ozone.apache.org/docs/edge/feature/streaming-write-pipeline.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14286. [Website v2] [Docs] [Core Concepts] Write pipelines [ozone-site]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #224:
URL: https://github.com/apache/ozone-site/pull/224#discussion_r2672693415
##########
docs/03-core-concepts/02-replication/02-write-pipelines.md:
##########
@@ -1,5 +1,298 @@
+---
+sidebar_label: Write Pipelines
+---
+
# Write Pipelines
-**TODO:** File a subtask under [HDDS-9857](https://issues.apache.org/jira/browse/HDDS-9857) and complete this page or section.
+Write pipelines are a fundamental component of Apache Ozone's storage architecture, enabling reliable data storage across distributed nodes. This document provides a comprehensive overview of write pipelines, covering both replication and erasure coding approaches, their architecture, implementation details, and usage patterns.
+
+## What are Write Pipelines?
+
+Write pipelines are groups of Datanodes that work together as a unit to store and replicate data in Ozone. They serve as the foundation for Ozone's data redundancy strategy, providing:
+
+- A coordinated path for write operations across multiple nodes
+- Consistency guarantees for data replication
+- Efficient management of data distribution and storage
+
+The Storage Container Manager (SCM) is responsible for creating and managing write pipelines, selecting appropriate Datanodes based on factors like availability, capacity, and network topology.
+
+## Pipeline Types
+
+Ozone supports different types of write pipelines to accommodate various durability and storage efficiency requirements:
+
+### 1. Ratis Pipelines (Replicated)
+
+Ratis pipelines use the [Apache Ratis](https://ratis.apache.org/) implementation of the Raft consensus protocol for strongly consistent replication.
+
+- **Structure**: Typically consists of three Datanodes (one leader, multiple followers)
+- **Consistency**: Provides strong consistency through synchronous replication
+- **Durability**: Data is fully replicated on all nodes in the pipeline
+- **Use Case**: Default replication strategy for most Ozone deployments
+
+
+
+#### Ratis Pipeline V1: Async API
+
+The original Ozone replication pipeline (V1) uses the Ratis Async API for data replication across multiple Datanodes:
+
+1. Client buffers data locally until a certain threshold is reached
+2. Data is sent to the leader Datanode in the pipeline
+3. Leader replicates data to follower Datanodes
+4. Once a quorum of Datanodes acknowledge the write, the operation is considered successful
+
+This approach ensures data consistency but has some limitations in terms of network topology awareness and buffer handling efficiency.
+
+#### Ratis Pipeline V2: Streaming API
Review Comment:
should we add some reference here https://www.cloudera.com/blog/technical/ozone-write-pipeline-v2-with-ratis-streaming.html?
or maybe other place, I just feel this article is so good and it has to be placed somewhere in the official site.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14286. [Website v2] [Docs] [Core Concepts] Write pipelines [ozone-site]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #224:
URL: https://github.com/apache/ozone-site/pull/224#discussion_r2675081109
##########
docs/03-core-concepts/02-replication/02-write-pipelines.md:
##########
@@ -1,5 +1,298 @@
+---
+sidebar_label: Write Pipelines
+---
+
# Write Pipelines
-**TODO:** File a subtask under [HDDS-9857](https://issues.apache.org/jira/browse/HDDS-9857) and complete this page or section.
+Write pipelines are a fundamental component of Apache Ozone's storage architecture, enabling reliable data storage across distributed nodes. This document provides a comprehensive overview of write pipelines, covering both replication and erasure coding approaches, their architecture, implementation details, and usage patterns.
+
+## What are Write Pipelines?
+
+Write pipelines are groups of Datanodes that work together as a unit to store and replicate data in Ozone. They serve as the foundation for Ozone's data redundancy strategy, providing:
+
+- A coordinated path for write operations across multiple nodes
+- Consistency guarantees for data replication
+- Efficient management of data distribution and storage
+
+The Storage Container Manager (SCM) is responsible for creating and managing write pipelines, selecting appropriate Datanodes based on factors like availability, capacity, and network topology.
+
+## Pipeline Types
+
+Ozone supports different types of write pipelines to accommodate various durability and storage efficiency requirements:
+
+### 1. Ratis Pipelines (Replicated)
+
+Ratis pipelines use the [Apache Ratis](https://ratis.apache.org/) implementation of the Raft consensus protocol for strongly consistent replication.
+
+- **Structure**: Typically consists of three Datanodes (one leader, multiple followers)
+- **Consistency**: Provides strong consistency through synchronous replication
+- **Durability**: Data is fully replicated on all nodes in the pipeline
+- **Use Case**: Default replication strategy for most Ozone deployments
+
+
+
+#### Ratis Pipeline V1: Async API
+
+The original Ozone replication pipeline (V1) uses the Ratis Async API for data replication across multiple Datanodes:
+
+1. Client buffers data locally until a certain threshold is reached
+2. Data is sent to the leader Datanode in the pipeline
+3. Leader replicates data to follower Datanodes
+4. Once a quorum of Datanodes acknowledge the write, the operation is considered successful
+
+This approach ensures data consistency but has some limitations in terms of network topology awareness and buffer handling efficiency.
+
+#### Ratis Pipeline V2: Streaming API
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org