[GitHub] [spark] gaborgsomogyi commented on a change in pull request #29729: [SPARK-32032][SS] Avoid infinite wait in driver because of KafkaConsumer.poll(long) API

[GitBox <gi...@apache.org>]

gaborgsomogyi commented on a change in pull request #29729:
URL: https://github.com/apache/spark/pull/29729#discussion_r499549277



##########
File path: docs/ss-migration-guide.md
##########
@@ -26,6 +26,19 @@ Note that this migration guide describes the items specific to Structured Stream
 Many items of SQL migration can be applied when migrating Structured Streaming to higher versions.
 Please refer [Migration Guide: SQL, Datasets and DataFrame](sql-migration-guide.html).
 
+## Upgrading from Structured Streaming 3.0 to 3.1
+
+- In Spark 3.0 and below, secure Kafka processing needed the following ACLs from driver perspective:
+  * Topic resource describe operation
+  * Topic resource read operation
+  * Group resource read operation
+
+  Since Spark 3.1, offsets are obtained with `AdminClient` instead of `KafkaConsumer` and now the following ACLs needed from driver perspective:
+  * Topic resource describe operation
+
+  Since `AdminClient` in driver is not connecting to consumer group, `group.id` based authorization will not work anymore (executors never done group based authorization).

Review comment:
       Some of the users authorizing based on `group.id` which will not work with this PR (topic based authorization is a good alternative). Nothing more, nothing less. If you can help how to make it more descriptive it's welcome.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org