You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/05/24 16:41:05 UTC
[GitHub] [spark] tashoyan commented on pull request #28623: [SPARK-31805][SS] Exclude group.id from consumer settings when using the assign strategy
tashoyan commented on pull request #28623:
URL: https://github.com/apache/spark/pull/28623#issuecomment-633257746
#### Consumers inside executors
Indeed, executors have their own Kafka consumers, and these consumers always have `group.id` set: [KafkaSourceProvider.kafkaParamsForExecutors()](https://github.com/apache/spark/blob/bcadd5c3096109878fe26fb0d57a9b7d6fdaa257/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L599). We can see group ids in executor logs, INFO level is enough.
However, the consumers inside executors always do `assign()` and never `subscribe()`. Then, there is no reason to set `group.id` for them. I would suggest to remove this setting for consumers.
Nevertheless, executors do not fail as driver does. My investigation is below in this post.
#### Kafka consumers behavior
By [KafkaConsumer specification](https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html), the `assign` mechanism does not use consumer groups at all. Contrarily to `subscribe`, `assign` assumes manual assignment of topic partitions.
If I am not missing anything, Structured Streaming manually assigns Spark executors to Kafka partitions. We do not use consumer groups at all, so we don't need them. Therefore, I am not sure why Structured Streaming provides the option "subscribe".
By not specifying `group.id`, we can avoid the authorization problem on a secure Kafka broker.
#### Weird behavior of consumers inside executors
A weird thing is that executors do not fail with GroupAuthorizationException as the driver does. The difference between driver's consumer and executor's consumer is following:
- driver sequence - see [KafkaOffsetReader](https://github.com/apache/spark/blob/v3.0.0-preview2/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala):
1. create consumer (using ConsumerStrategy)
1. `subscribe` or `assign` as per ConsumingStrategy
1. run `consumer.poll()` - [KafkaOffsetReader.fetchTopicPartitions()](https://github.com/apache/spark/blob/bcadd5c3096109878fe26fb0d57a9b7d6fdaa257/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala#L116)
- executor sequence - see [KafkaDataConsumer](https://github.com/apache/spark/blob/v3.0.0-preview2/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala):
1. create consumer
1. always `assign` (no strategy)
1. make a number of seeks, e.g. `seekToBeginning()`, `seekToEnd()` - see for example [InternalKafkaConsumer.getAvailableOffsetRange()](https://github.com/apache/spark/blob/bcadd5c3096109878fe26fb0d57a9b7d6fdaa257/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala#L103).
1. run `consumer.poll()` - [InternalKafkaConsumer.fetchData()](https://github.com/apache/spark/blob/bcadd5c3096109878fe26fb0d57a9b7d6fdaa257/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala#L74)
These seek operations make the difference. I made a trivial application with KafkaConsumer - if I insert `seekToBeginning()` before calling `poll()`, the consumer successfully reads records without facing GroupAuthorizationException. Maybe it is a Kafka bug (I tried with Kafka 2.2.0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org