You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "GuoHao (Jira)" <ji...@apache.org> on 2023/10/27 09:57:00 UTC
[jira] [Comment Edited] (HDDS-8691) Ozone Streaming Write Pipeline Phase 2
[ https://issues.apache.org/jira/browse/HDDS-8691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17780290#comment-17780290 ]
GuoHao edited comment on HDDS-8691 at 10/27/23 9:56 AM:
--------------------------------------------------------
Hi [~ivanandika] ,Thank you for your interest and testing of ozone streaming features!
1. What is the size of the test file you are using?
2. We recommend you to use [https://github.com/szetszwo/ozone-benchmark/blob/master/benchmark-conf.xml]
The version I use is different from the community version, and I've added the following changes:
1. ratis sends a raft async rpc when stream closes, which is used to grow the index of the raft log, which is used to update the bcsid in the put block.
2. since stream is strongly consistent, the bcsid can actually be omitted if the cluster always uses stream to transfer data, so we changed this part of the logic in ratis to no longer send a raft async rpc, and replaced the bcsid in the put block with a 0.
3. The production environment has been running for about 1 year and no data integrity issues have been found.
A few months ago I [~szetszwo] and [~sammichen] discussed the ozone streaming feature, and didn't find any problems with the change, but it's a big change and needs to be discussed with other members of the community, so if you're interested we can work together to move it forward!
was (Author: nicholas niu):
Hi [~ivanandika] ,Thank you for your interest and testing of ozone streaming features!
1. What is the size of the test file you are using?
2. We recommend you to use https://github.com/szetszwo/ozone-benchmark/blob/master/benchmark-conf.xml
The version I use online is different from the community version, and I've added the following changes:
1. ratis sends a raft async rpc when stream closes, which is used to grow the index of the raft log, which is used to update the bcsid in the put block.
2. since stream is strongly consistent, the bcsid can actually be omitted if the cluster always uses stream to transfer data, so we changed this part of the logic in ratis to no longer send a raft async rpc, and replaced the bcsid in the put block with a 0.
3. The production environment has been running for about 1 year and no data integrity issues have been found.
A few months ago I [~szetszwo] and [~sammichen] discussed the ozone streaming feature, and didn't find any problems with the change, but it's a big change and needs to be discussed with other members of the community, so if you're interested we can work together to move it forward!
> Ozone Streaming Write Pipeline Phase 2
> --------------------------------------
>
> Key: HDDS-8691
> URL: https://issues.apache.org/jira/browse/HDDS-8691
> Project: Apache Ozone
> Issue Type: New Feature
> Reporter: Tsz-wo Sze
> Assignee: Tsz-wo Sze
> Priority: Major
>
> This is the umbrella JIRA for the subtasks in Ozone Streaming Write Pipeline Phase 2.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org