[jira] [Created] (SPARK-27002) Latest kafka delegation token not always picked up

["Gabor Somogyi (JIRA)" <ji...@apache.org>]

Gabor Somogyi created SPARK-27002:
-------------------------------------

             Summary: Latest kafka delegation token not always picked up
                 Key: SPARK-27002
                 URL: https://issues.apache.org/jira/browse/SPARK-27002
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 3.0.0
            Reporter: Gabor Somogyi


Lately KAFKA-8004 has been discovered and this triggered me to do further tests with Spark.
Namely I've created long running Kafka to Kafka tests on 4 node cluster with randomly thrown artificial exceptions.

Test scenario:
* 4 node cluster
* Yarn
* Kafka broker version 2.1.0
* security.protocol = SASL_SSL
* sasl.mechanism = SCRAM-SHA-512

Kafka broker settings:
* delegation.token.expiry.time.ms=600000 (10 min)
* delegation.token.max.lifetime.ms=1200000 (20 min)
* delegation.token.expiry.check.interval.ms=300000 (5 min)

After each 7.5 minutes new delegation token obtained from Kafka broker (10 min * 0.75).
But when token expired after 10 minutes (Spark obtains new one the doesn't renew the old), the brokers expiring thread comes after each 5 minutes (invalidates expired tokens) and artificial exception has been thrown inside the Spark application, then the latest delegation token not always picked up.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org