You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ryan Blough (Jira)" <ji...@apache.org> on 2026/05/28 00:18:00 UTC

[jira] [Created] (HDDS-15403) Build a second EC reconstruction procedure to target faster time to recovery

Ryan Blough created HDDS-15403:
----------------------------------

             Summary: Build a second EC reconstruction procedure to target faster time to recovery
                 Key: HDDS-15403
                 URL: https://issues.apache.org/jira/browse/HDDS-15403
             Project: Apache Ozone
          Issue Type: Improvement
          Components: EC, Ozone Datanode
            Reporter: Ryan Blough


This Jira is to track and implement a different erasure coding reconstruction procedure with a focus on completing the reconstruction task faster.

The EC reconstruction procedure we currently use is sequential, by chunk, and has a minimal resource footprint. The core loop of the reconstruction is:
 # Fetch a container chunk over the network.
 # Load the chunk into off-heap memory.
 # Do reconstruction on the chunk.
 # Write that chunk over the network to the target nodes.
 # On confirmation of write, iterate the loop.

This has the advantage of consuming minimal resources at each step, with memory footprint being limited to one container chunk size, and consuming a single thread.

However, it is also has network at either end of a loop that iterates many times.

The concept of this second reconstruction method is to complete steps 1-4 in a single stage each. The tradeoff will be faster time to recovery for the individual container in exchange for a larger resource footprint (namely enough memory to store the full-size container).

After a first pass to establish end-to-end single-threaded behavior in comparison with the loop, additional considerations are likely to include async, multithreading, and revisiting the single-chunk work unit depending on how conventional Reed-Solomon (the algorithm in libhadoop) scales with work unit size.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org