[jira] [Created] (HDDS-11308) ClassCastException during simultaneous putBlock operations on an EC container

["Ethan Rose (Jira)" <ji...@apache.org>]

Ethan Rose created HDDS-11308:
---------------------------------

             Summary: ClassCastException during simultaneous putBlock operations on an EC container
                 Key: HDDS-11308
                 URL: https://issues.apache.org/jira/browse/HDDS-11308
             Project: Apache Ozone
          Issue Type: Bug
          Components: EC, Ozone Datanode
            Reporter: Ethan Rose


h2. Issue

While writing EC data the following exception was observed:
{code:java}
2024-07-30 10:42:22,547 WARN [ChunkReader-8]-org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler: Operation: PutBlock , Trace ID:  , Message: java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode , Result: CONTAINER_INTERNAL_ERROR , StorageContainerException Occurred.
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode
	at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:228)
	at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:328)
	at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.lambda$dispatch$0(HddsDispatcher.java:176)
	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
	at org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:175)
	at org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:57)
	at org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:50)
	at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
	at org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
	at org.apache.hadoop.hdds.tracing.GrpcServerInterceptor$1.onMessage(GrpcServerInterceptor.java:49)
	at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:333)
	at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:316)
	at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:835)
	at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode
	at java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1859)
	at java.util.HashMap$TreeNode.putTreeVal(HashMap.java:2038)
	at java.util.HashMap.putVal(HashMap.java:639)
	at java.util.HashMap.put(HashMap.java:613)
	at java.util.HashSet.add(HashSet.java:220)
	at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.addToPendingPutBlockCache(KeyValueContainer.java:831)
	at org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.persistPutBlock(BlockManagerImpl.java:209)
	at org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.putBlock(BlockManagerImpl.java:103)
	at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handlePutBlock(KeyValueHandler.java:551)
	at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.dispatchRequest(KeyValueHandler.java:254)
	at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:225)
	... 17 more
2024-07-30 10:42:22,550 WARN [ChunkReader-8]-org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer: Moving container /data/...current/containerDir189/227982 to state UNHEALTHY from state:OPEN
{code}
Container 227982 is an EC container, and Ozone was later able to restore all healthy replicas using offline reconstruction.
h2. Cause

There is a putBlock cache for tracking updates to the block count metadata of an open container. The cache is not thread safe so when two threads access it at the same time we may get undefined behavior like the exception shown above.
h3. Explanation

The cache is implemented as a non-concurrent {{{}HashSet{}}}. The [code comments acknowledge this|https://github.com/apache/ozone/blob/98369a8343f12479af99db4c83696945d40c1e3d/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java#L112] and assume only one instance of putBlock per container will run at a time. This assumption that only one putBlock per container will run at a time is also assumed by the metadata updates done by putBlock, which increment counters in RocksDB and can have incorrect values if two updates happen at the same time (time of check time of use, see HDDS-8129 causing issues like HDDS-5359)

This assumptions were probably in place before EC was added. For Ratis, the {{ContainerStateMachine}} enforces that only one putBlock per container will run at a time. For EC, however, there is no such enforcement. EC write requests go through {{XceiverServerGrpc}} to {{GrpcXceiverService}} where they are handled by a [thread pool|https://github.com/apache/ozone/blob/9ba4a73f46156cae31c6c074290549fdb95e6f2e/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerGrpc.java#L149] that assumes it is only handling reads. If two EC clients issue putBlock for different blocks in the same container, we may see:
# Incorrect metadata counters.
# Exceptions like the one above that mark the container unhealthy and fail the write.

Based on inspection of the {{HashSet}} code, I believe this specific exception will only occur if the two concurrent putBlock IDs hash to the same bin in the set. This explains why the issue has existed for about 2 years but have just seen it for the first time now.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org