You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ethan Rose (Jira)" <ji...@apache.org> on 2021/10/20 20:37:11 UTC
[jira] [Updated] (HDDS-2010) PipelineID management for multi-raft,
in SCM or in datanode?
[ https://issues.apache.org/jira/browse/HDDS-2010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ethan Rose updated HDDS-2010:
-----------------------------
Target Version/s: 1.3.0 (was: 1.2.0)
I am managing the 1.2.0 release and we currently have more than 600 issues targeted for 1.2.0. I am moving the target field to 1.3.0.
If you are actively working on this jira and believe this should be targeted for the 1.2.0 release, Please reach out to me via Apache email or Slack.
> PipelineID management for multi-raft, in SCM or in datanode?
> ------------------------------------------------------------
>
> Key: HDDS-2010
> URL: https://issues.apache.org/jira/browse/HDDS-2010
> Project: Apache Ozone
> Issue Type: New Feature
> Components: Ozone Datanode
> Reporter: Li Cheng
> Assignee: Li Cheng
> Priority: Major
> Labels: TriagePending
>
> With the intention to support multi-raft, I wanna bring up a question on how the pipeline unique ids be managed. Since every datanode can be member in multiple raft pipelines, the pipeline ids need to be persisted with the datanode for recovery purpose (we can talk about recovery later). Generally there are two options:
> # Store in datanode (like datanodeDetails) and every time pipelines mapping change on single datanode, pipeline ids will be serialized to local file. This way will lead to many more local serialization of things like datanodeDetails, but the updates are only for local datanode change. Improvement can be made like linking a serializable object to datanodeDetails and datanode keeps updating the new pipeline ids to the serializable object instead the details file. On the other hand, since the pipeline ids are stored only in datanode locally, there will be no global view in SCM. (or we can store a lazy copy?)
> # Stored in SCM. SCM can maintain a large mapping between datanode ids and pipeline ids. But this way will lead to an exponentially increasing frequency in SCM updates since the pipeline mapping changes are way more complex and happen all the time. Obviously this gives SCM too much pressure, but it can also give SCM a global view on the management over datanodes and multi raft pipelines.
>
> Thoughts? [~xyao] [~Sammi]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org