You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Zhen Wang (Jira)" <ji...@apache.org> on 2023/02/22 03:02:00 UTC
[jira] [Created] (HBASE-27658) HBase token expired because the connection was not released
Zhen Wang created HBASE-27658:
---------------------------------
Summary: HBase token expired because the connection was not released
Key: HBASE-27658
URL: https://issues.apache.org/jira/browse/HBASE-27658
Project: HBase
Issue Type: Bug
Components: spark
Environment: spark: 3.1.1
hbase-connector: 1.0.1-SNAPSHOT
Reporter: Zhen Wang
I used hbase-spark to read hbase data in spark streaming application, and the token expired error occurred after 7 days.
Error:
{code:java}
23/02/20 19:29:07 INFO AsyncRequestFutureImpl: id=38643, table=XXX, attempt=11/16, failureCount=1ops, last exception=java.io.IOException: Call to XXX.XXX/XXX.XXX.XXX.XXX:60020 failed on local exception: java.io.IOException: org.apache.hbase.thirdparty.io.netty.handler.codec.DecoderException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Token has expired on XXX.XXX,60020,1675839912342, tracking started null, retrying after=10009ms, operationsToReplay=1 {code}
HBase RpcClient has expired connections that have not been cleaned up.
arthas command:
watch org.apache.hadoop.hbase.ipc.AbstractRpcClient getConnection "target.connections.values.\{token.toString}" -x 2 -n 2
result:
{code:java}
@ArrayList[
@String[Kind: HBASE_AUTH_TOKEN, Service: a88deda4-d74c-483e-af8b-0fb9de7c7594, Ident: ((username=XXX, keyId=575, issueDate=1676858059133, expirationDate=1677462859133, sequenceNumber=772469))],
...
@String[Kind: HBASE_AUTH_TOKEN, Service: a88deda4-d74c-483e-af8b-0fb9de7c7594, Ident: ((username=XXX, keyId=573, issueDate=1676656459093, expirationDate=1677261259093, sequenceNumber=587447))],
...
@String[Kind: HBASE_AUTH_TOKEN, Service: a88deda4-d74c-483e-af8b-0fb9de7c7594, Ident: ((username=XXX, keyId=571, issueDate=1676454859019, expirationDate=1677059659019, sequenceNumber=430605))],
...
]{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)