You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Ronald Macmaster (Jira)" <ji...@apache.org> on 2026/10/20 17:14:00 UTC
[jira] [Created] (HDFS-18006) DFSClient: Shared Memory Leak when REQUEST_SHORT_CIRCUIT_FDS fails
Ronald Macmaster created HDFS-18006:
---------------------------------------
Summary: DFSClient: Shared Memory Leak when REQUEST_SHORT_CIRCUIT_FDS fails
Key: HDFS-18006
URL: https://issues.apache.org/jira/browse/HDFS-18006
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.5.0, 3.4.1, 3.6.0
Environment: client: hadoop 3.4.1 (HBase 2.5.11 Regionserver)
server: hadoop 3.4.2
Reporter: Ronald Macmaster
When the DFSClient {{BlockReaderFactory#createShortCircuitReplicaInfo}} initializes shared memory slots for Short Circuit Reads, it doesn't properly clean it up it if an unexpected error is encountered. This leaks shared memory on the DataNode host over a prolonged period of time.
The problem is below in {{{}BlockReaderFactory#requestFileDescriptors{}}}. The shmem slot is only free'd on {{ERROR_ACCESS_TOKEN}} (HDFS-16198) or when it throws an {{{}IOException{}}}, ({{{}createShortCircuitReplicaInfo{}}} catches it above).
Three other {{requestFileDescriptors}} cases can return {{null}} without freeing the slot:
* {{SUCCESS}}
* {{ERROR_UNSUPPORTED}}
* {{{}default{}}}, which handles general {{ERRORresponses from the DataNode.}}
Even though the DataNode unregisters its side of the slot in {{DataXceiver#requestShortCircuitFds}} on failure, it cannot fully release the memory until the client closes its end as well!
h2. Context
We discovered this issue while experimenting with machine decom/recom on our HBase clusters. If a DataNode rejoins the cluster without previously decomming properly, it can trigger a mass block deletions and ReplicaNotFoundExceptions on the HBase Regionserver's DFSClient (which caches the old block locations).
The {{DataNode}} handles {{ReplicaNotFoundException}} by returning an ERROR response to the client and this causes it to temporarily disable short circuit reads. We curiously tuned down dfs.domain.socket.disable.interval.seconds to 0 to see if we could bypass this otherwise harmless error (the read is retried over TCP), but encountered a shared memory leak at scale.
{code:java}
WARN impl.BlockReaderFactory: BlockReaderFactory(fileName=..., block=...): unknown response code ERROR while attempting to set up short-circuit access. No data exists for block BP-...:blk_... . Short-circuit read for DataNode DatanodeInfoWithStorage[...] is disabled temporarily for 600 seconds based on dfs.domain.socket.disable.interval.seconds.
{code}
{code:java}
WARN [RpcServer.default.FPRWQ.Fifo.read.handler=157,queue=16,port=16022] impl.BlockReaderFactory: I/O error constructing remote block reader for block BP-..:blk_..
java.io.IOException: Got error, status=ERROR, status message opReadBlock BP-..:blk_.. received exception org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not found for BP-..:blk_... The block may have been removed recently by the balancer or by intentionally reducing the replication factor. This condition is usually harmless. To be certain, please check the preceding datanode log messages for signs of a more serious issue., for OP_READ_BLOCK, self=/<ip>:<port>, remote=/<ip>:<port>, for file /hbase/data/<ns>/<table>/<region>/<col>/<hfile>, for pool BP-.. block ..
at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:128) ~[hadoop-hdfs-client-3.4.1.jar:?]
at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:104) ~[hadoop-hdfs-client-3.4.1.jar:?]
at org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.checkSuccess(BlockReaderRemote.java:452) ~[hadoop-hdfs-client-3.4.1.jar:?]
at org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.newBlockReader(BlockReaderRemote.java:420) ~[hadoop-hdfs-client-3.4.1.jar:?]
at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:861) ~[hadoop-hdfs-client-3.4.1.jar:?]
at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:757) ~[hadoop-hdfs-client-3.4.1.jar:?]
at org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:381) ~[hadoop-hdfs-client-3.4.1.jar:?]
at org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:715) ~[hadoop-hdfs-client-3.4.1.jar:?]
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org