[GitHub] [spark] gaborgsomogyi opened a new pull request #24305: [SPARK-27294][SS] Add multi-cluster Kafka delegation token

[GitBox <gi...@apache.org>]

gaborgsomogyi opened a new pull request #24305: [SPARK-27294][SS] Add multi-cluster Kafka delegation token
URL: https://github.com/apache/spark/pull/24305
 
 
   ## What changes were proposed in this pull request?
   
   The actual implementation doesn't support multi-cluster Kafka connection with delegation token. In this PR I've added this functionality.
   
   What this PR contains:
   * New way of configuration
   * Multiple delegation token obtain/store/use functionality
   * Temporary turn off DStreams delegation token usage not to create horror big PR
   
   What this PR doesn't contain:
   * DStreams implementation
   * Documentation
   
   ## How was this patch tested?
   
   Existing + additional unit tests.
   Additionally tested on cluster.
   
   Test scenario:
   
   * 2 * 4 node clusters
   * The 4-4 nodes are in different kerberos realms
   * Cross-Realm trust between the 2 realms
   * Yarn
   * Kafka broker version 2.1.0
   * security.protocol = SASL_SSL
   * sasl.mechanism = SCRAM-SHA-512
   * Artificial exceptions during processing
   * Source reads from realm1 sink writes to realm2
   
   Kafka broker settings:
   
   * delegation.token.expiry.time.ms=600000 (10 min)
   * delegation.token.max.lifetime.ms=1200000 (20 min)
   * delegation.token.expiry.check.interval.ms=300000 (5 min)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org