[jira] [Resolved] (SPARK-41073) Spark ThriftServer generate huge amounts of DelegationToken

["zhengchenyu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-41073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

zhengchenyu resolved SPARK-41073.
---------------------------------
    Resolution: Duplicate

> Spark ThriftServer generate huge amounts of DelegationToken
> -----------------------------------------------------------
>
>                 Key: SPARK-41073
>                 URL: https://issues.apache.org/jira/browse/SPARK-41073
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.0.1
>            Reporter: zhengchenyu
>            Priority: Major
>         Attachments: SPARK-41073.proposal.A.draft.001.patch
>
>
> In our cluster, zookeeper nearly crashed. I found the znodes of /zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot increased quickly. 
> After some research, I found some sql running on spark-thriftserver obtain huge amounts of DelegationToken.
> The reason is that in these spark-sql, every hive parition acquire a different delegation token. 
> And HadoopRDDs in thriftserver can't share credentials from CoarseGrainedSchedulerBackend::delegationTokens, we must share it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org