You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Rajan Dhabalia (Jira)" <ji...@apache.org> on 2026/09/05 00:38:00 UTC
[jira] [Created] (HDFS-17976) HDFS DataNode add configurable inactivity-based timeout for DataNode block transfers
Rajan Dhabalia created HDFS-17976:
-------------------------------------
Summary: HDFS DataNode add configurable inactivity-based timeout for DataNode block transfers
Key: HDFS-17976
URL: https://issues.apache.org/jira/browse/HDFS-17976
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs
Reporter: Rajan Dhabalia
h2. Summary
Add a configurable inactivity-based timeout to safely terminate stalled DataNode block transfers, reclaim resources, and preserve partially written replicas for recovery.
h2. Issue Type
Improvement
h2. Component/s
datanode
h2. Description
h3. Motivation
A block write remains active until the client sends the final packet and the block is finalized. If a client crashes, loses connectivity, or becomes unresponsive, the DataNode transfer thread can remain blocked waiting for additional data.
Stalled transfers can accumulate over time, consuming transfer threads and other resources and potentially affecting healthy client operations.
h3. Approach
Introduce a configurable inactivity timeout for DataNode block transfers.
When enabled:
* Track packet-arrival activity for each ongoing block write.
* Schedule an inactivity check for each active transfer.
* Terminate the transfer if no packet is received within the configured timeout.
* Flush buffered data before termination.
* Close the input stream to interrupt blocked reads.
* Keep the replica in *RBW (Replica Being Written)* state.
* Allow existing NameNode lease recovery and block synchronization to recover and finalize the replica.
The timeout uses a shared `ScheduledExecutorService` and is created lazily only when enabled. When disabled, there are no additional scheduler threads or timeout processing.
h3. Configuration
||Property||Default||Description||
|`dfs.datanode.last.packet.receive.timeout.ms`|0|Inactivity timeout for an ongoing block transfer. If no packet is received within the configured window, the transfer is considered stalled and terminated. `0` or negative disables the feature.|
h3. Transfer Lifecycle
{code:java}
Client starts block write
|
v
DataNode receives packets
|
v
Track packet activity
|
v
Packet received?
/ \
Yes No
| |
v v
Reset timer Timeout reached
|
v
Flush buffered data
|
v
Close input stream
|
v
Replica remains RBW
|
v
Existing lease recovery /
block synchronization
|
v
Block finalized
{code}
h3. Data Safety
When a stalled transfer is terminated:
# Buffered data is flushed.
# The input stream is closed.
# The replica remains in RBW state.
# Existing HDFS recovery mechanisms handle subsequent recovery/finalization.
This preserves data already received by the DataNode while reclaiming stalled transfer resources.
h3. Results
The inactivity timeout provides:
* Bounded resource usage for stalled transfers.
* Prevention of indefinite transfer-thread retention after client failures.
* Reduced risk of transfer-thread exhaustion during client failure bursts.
* Preservation of partially written replica data.
* Compatibility with existing HDFS recovery mechanisms.
* No additional overhead when disabled.
h3. Estimated Impact
This is primarily a reliability and resource-reclamation improvement rather than a throughput optimization.
||Dimension||Without Timeout||With Inactivity Timeout||
|Stalled transfer lifetime|Potentially unbounded|Bounded by configured timeout|
|Transfer threads|Can accumulate|Reclaimed after inactivity|
|Client failure bursts|Risk of resource exhaustion|Resource usage remains bounded|
|Partially written data|Preserved through existing recovery|Preserved; replica remains RBW|
|Normal transfers|Existing behavior|Unchanged|
|Disabled overhead|Existing behavior|No additional scheduler/timeout processing|
h3. Operational Considerations
The timeout should be configured based on workload characteristics. An overly aggressive timeout could terminate legitimately slow transfers, while a sufficiently large timeout allows temporary network or client stalls while still reclaiming resources from genuinely stalled transfers.
The feature is therefore *disabled by default* and can be enabled explicitly by operators.
h2. Expected Impact
* Improve DataNode resilience to crashed, disconnected, or hung clients.
* Prevent stalled transfers from holding resources indefinitely.
* Bound transfer-resource usage during client failure bursts.
* Preserve partially written data for existing recovery mechanisms.
* Keep normal block-transfer behavior unchanged when disabled.
h2. Backward Compatibility
The feature is {*}disabled by default{*}. Existing DataNode block-transfer behavior remains unchanged unless `dfs.datanode.last.packet.receive.timeout.ms` is explicitly configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org