[jira] [Created] (HDDS-15799) OPEN RATIS pipelines retain stale DatanodeDetails without RATIS_DATASTREAM after DataStream is enabled

["Yashaswini G A (Jira)" <ji...@apache.org>]

Yashaswini G A created HDDS-15799:
-------------------------------------

             Summary: OPEN RATIS pipelines retain stale DatanodeDetails without RATIS_DATASTREAM after DataStream is enabled
                 Key: HDDS-15799
                 URL: https://issues.apache.org/jira/browse/HDDS-15799
             Project: Apache Ozone
          Issue Type: Bug
          Components: Ozone Datanode
            Reporter: Yashaswini G A


Enabling Ratis DataStream on an existing cluster can leave OPEN pipelines whose member DatanodeDetails do not include {{{}RATIS_DATASTREAM{}}}, even after datanodes are listening on the DataStream port and SCM’s node registry shows the port correctly. O3FS/Ozone client DataStream writes then fail at runtime.

When {{ozone.fs.datastream.enabled=true}} and a write exceeds the auto-threshold, the client uses BlockDataStreamOutput, which requires every pipeline member to have an explicit {{RATIS_DATASTREAM}} port (hasPort() — no fallback to the RATIS port).

Pipeline members are a frozen snapshot of DatanodeDetails captured at pipeline creation. SCM does not refresh member ports when:
 * {{dfs.container.ratis.datastream.enabled}} is enabled after pipelines were created, or

 * datanodes later gain the {{RATIS_DATASTREAM}} port.

SCM can still allocate blocks/containers on these OPEN pipelines. The client fails with:
 
 {{java.io.IOException: RATIS_DATASTREAM port is missing for datanode f475cb4a-f60a-4437-b102-d317b8e6118c(ccycloud-4.quasar-fqjoax.root.comops.site/10.140.80.132) in pipeline Pipeline-05ad9496-a228-4644-ab9f-ea6dc0426d73; datastream is disabled for this pipeline"}}
 
 
h3. Steps to reproduce
 # Start cluster with default configs {{dfs.container.ratis.datastream.enabled=false}} and {{ozone.fs.datastream.enabled=false}}

 # SCM may have created OPEN RATIS pipelines (RF=3).

 # Enable datastream configs in step 1 and restart ozone service.

 # Confirm DNs listen on DataStream port and SCM datanode list shows {{{}RATIS_DATASTREAM{}}}.

 # Perform O3FS write > {{ozone.fs.datastream.auto.threshold}} (e.g. 20 MB {{ozone fs -put}} / {{{}O3fs.put{}}}).

h3. Actual behavior

Write fails if block allocation uses a pre-existing OPEN pipeline whose members lack {{{}RATIS_DATASTREAM{}}}.
h3. Expected behavior

Either:
 * OPEN pipelines are refreshed/recreated so members include current ports, or

 * SCM/OM avoids allocating DataStream writes to incompatible pipelines, or

 * A clear upgrade path automatically closes stale OPEN RATIS pipelines when DataStream is enabled.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org