You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Rajan Dhabalia (Jira)" <ji...@apache.org> on 2026/09/05 00:38:00 UTC

[jira] [Created] (HDFS-17976) HDFS DataNode add configurable inactivity-based timeout for DataNode block transfers

Rajan Dhabalia created HDFS-17976:
-------------------------------------

             Summary: HDFS DataNode add configurable inactivity-based timeout for DataNode block transfers
                 Key: HDFS-17976
                 URL: https://issues.apache.org/jira/browse/HDFS-17976
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: hdfs
            Reporter: Rajan Dhabalia


h2. Summary

Add a configurable inactivity-based timeout to safely terminate stalled DataNode block transfers, reclaim resources, and preserve partially written replicas for recovery.
h2. Issue Type

Improvement
h2. Component/s

datanode
h2. Description
h3. Motivation

A block write remains active until the client sends the final packet and the block is finalized. If a client crashes, loses connectivity, or becomes unresponsive, the DataNode transfer thread can remain blocked waiting for additional data.

Stalled transfers can accumulate over time, consuming transfer threads and other resources and potentially affecting healthy client operations.
h3. Approach

Introduce a configurable inactivity timeout for DataNode block transfers.

When enabled:
 * Track packet-arrival activity for each ongoing block write.
 * Schedule an inactivity check for each active transfer.
 * Terminate the transfer if no packet is received within the configured timeout.
 * Flush buffered data before termination.
 * Close the input stream to interrupt blocked reads.
 * Keep the replica in *RBW (Replica Being Written)* state.
 * Allow existing NameNode lease recovery and block synchronization to recover and finalize the replica.

The timeout uses a shared `ScheduledExecutorService` and is created lazily only when enabled. When disabled, there are no additional scheduler threads or timeout processing.
h3. Configuration
||Property||Default||Description||
|`dfs.datanode.last.packet.receive.timeout.ms`|0|Inactivity timeout for an ongoing block transfer. If no packet is received within the configured window, the transfer is considered stalled and terminated. `0` or negative disables the feature.|
h3. Transfer Lifecycle
{code:java}
Client starts block write
        |
        v
DataNode receives packets
        |
        v
Track packet activity
        |
        v
Packet received?
   /            \
 Yes             No
  |               |
  v               v
Reset timer    Timeout reached
                  |
                  v
          Flush buffered data
                  |
                  v
          Close input stream
                  |
                  v
          Replica remains RBW
                  |
                  v
       Existing lease recovery /
       block synchronization
                  |
                  v
            Block finalized
{code}
h3. Data Safety

When a stalled transfer is terminated:
 # Buffered data is flushed.
 # The input stream is closed.
 # The replica remains in RBW state.
 # Existing HDFS recovery mechanisms handle subsequent recovery/finalization.

This preserves data already received by the DataNode while reclaiming stalled transfer resources.
h3. Results

The inactivity timeout provides:
 * Bounded resource usage for stalled transfers.
 * Prevention of indefinite transfer-thread retention after client failures.
 * Reduced risk of transfer-thread exhaustion during client failure bursts.
 * Preservation of partially written replica data.
 * Compatibility with existing HDFS recovery mechanisms.
 * No additional overhead when disabled.

h3. Estimated Impact

This is primarily a reliability and resource-reclamation improvement rather than a throughput optimization.
||Dimension||Without Timeout||With Inactivity Timeout||
|Stalled transfer lifetime|Potentially unbounded|Bounded by configured timeout|
|Transfer threads|Can accumulate|Reclaimed after inactivity|
|Client failure bursts|Risk of resource exhaustion|Resource usage remains bounded|
|Partially written data|Preserved through existing recovery|Preserved; replica remains RBW|
|Normal transfers|Existing behavior|Unchanged|
|Disabled overhead|Existing behavior|No additional scheduler/timeout processing|
h3. Operational Considerations

The timeout should be configured based on workload characteristics. An overly aggressive timeout could terminate legitimately slow transfers, while a sufficiently large timeout allows temporary network or client stalls while still reclaiming resources from genuinely stalled transfers.

The feature is therefore *disabled by default* and can be enabled explicitly by operators.
h2. Expected Impact
 * Improve DataNode resilience to crashed, disconnected, or hung clients.
 * Prevent stalled transfers from holding resources indefinitely.
 * Bound transfer-resource usage during client failure bursts.
 * Preserve partially written data for existing recovery mechanisms.
 * Keep normal block-transfer behavior unchanged when disabled.

h2. Backward Compatibility

The feature is {*}disabled by default{*}. Existing DataNode block-transfer behavior remains unchanged unless `dfs.datanode.last.packet.receive.timeout.ms` is explicitly configured.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org