You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Max Xie (Jira)" <ji...@apache.org> on 2022/02/11 11:24:00 UTC

[jira] [Created] (HDFS-16455) RBF: Router should explicitly specify the value of `jute.maxbuffer` in hadoop configuration files like core-site.xml

Max  Xie created HDFS-16455:
-------------------------------

             Summary: RBF: Router should explicitly specify the value of `jute.maxbuffer` in hadoop configuration files like core-site.xml
                 Key: HDFS-16455
                 URL: https://issues.apache.org/jira/browse/HDFS-16455
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: rbf
    Affects Versions: 3.3.0, 3.4.0
            Reporter: Max  Xie


Based on the current design for delegation token in secure Router, the total number of  tokens store and update in zookeeper using ZKDelegationTokenManager.  

But the default value of  system property `jute.maxbuffer` is just 4MB,  if Router store too many tokens in zk, it will throw  IOException   `{{{}Packet lenxx is out of range{}}}` and all Router will crash. 

 

In our cluster,  Routers crashed because of it. The crash logs are below 
2022-02-09 02:15:51,607 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Token renewal for identifier: (token for xxx: HDFS_DELEGATION_TOKEN owner=xxx/scheduler, renewer=hadoop, realUser=, issueDate=1644344146305, maxDate=1644948946305, sequenceNumbe
r=27136070, masterKeyId=1107); total currentTokens 279548  2022-02-09 02:16:07,632 WARN org.apache.zookeeper.ClientCnxn: Session 0x1000172775a0012 for server zkurl:2181, unexpected e
rror, closing socket connection and attempting reconnect
java.io.IOException: Packet len4194553 is out of range!
        at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:113)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145)
2022-02-09 02:16:07,733 WARN org.apache.hadoop.ipc.Server: IPC Server handler 1254 on default port 9001, call Call#144 Retry#0 org.apache.hadoop.hdfs.protocol.ClientProtocol.getDelegationToken from ip:46534
java.lang.RuntimeException: Could not increment shared counter !!
        at org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.incrementDelegationTokenSeqNum(ZKDelegationTokenSecretManager.java:582)
{{}}

When we restart a Router, it crashed again
{code:java}
2022-02-09 03:14:17,308 INFO org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager: Starting to load key cache.
2022-02-09 03:14:17,310 INFO org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager: Loaded key cache.
2022-02-09 03:14:32,930 WARN org.apache.zookeeper.ClientCnxn: Session 0x205584be35b0001 for server zkurl:2181, unexpected
error, closing socket connection and attempting reconnect
java.io.IOException: Packet len4194478 is out of range!
        at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:113)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145)

2022-02-09 03:14:33,030 ERROR org.apache.hadoop.hdfs.server.federation.router.security.token.ZKDelegationTokenSecretManagerImpl: Error starting threads for z
kDelegationTokens
java.io.IOException: Could not start PathChildrenCache for tokens {code}
Finnally, we config `-Djute.maxbuffer=10000000` in hadoop-env,sh to fix this issue.

After dig it, we found the number of the  znode `/ZKDTSMRoot/ZKDTSMTokensRoot`'s children node was more than 250000, which's data size was over 4MB.

 

Maybe we should  explicitly specify the value of `jute.maxbuffer` in hadoop configuration files like core-site.xml, hdfs-rbf-site.xml to configure a larger value 

 

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org