You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Cody Koeninger (JIRA)" <ji...@apache.org> on 2018/11/26 17:16:00 UTC
[jira] [Resolved] (SPARK-26121) [Structured Streaming] Allow users
to define prefix of Kafka's consumer group (group.id)
[ https://issues.apache.org/jira/browse/SPARK-26121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cody Koeninger resolved SPARK-26121.
------------------------------------
Resolution: Fixed
Assignee: Anastasios Zouzias
Fix Version/s: 3.0.0
resolved by pr https://github.com/apache/spark/pull/23103
> [Structured Streaming] Allow users to define prefix of Kafka's consumer group (group.id)
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-26121
> URL: https://issues.apache.org/jira/browse/SPARK-26121
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 2.4.0
> Reporter: Anastasios Zouzias
> Assignee: Anastasios Zouzias
> Priority: Minor
> Fix For: 3.0.0
>
>
> I run in the following situation with Spark Structure Streaming (SS) using Kafka.
>
> In a project that I work on, there is already a secured Kafka setup where ops can issue an SSL certificate per "[group.id|http://group.id/]", which should be predefined (or its prefix to be predefined).
>
> On the other hand, Spark SS fixes the [group.id|http://group.id/] to
>
> val uniqueGroupId = s"spark-kafka-source-${UUID.randomUUID}-${metadataPath.hashCode}"
>
> see, i.e.,
>
> [https://github.com/apache/spark/blob/v2.4.0/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L124]
>
> https://github.com/apache/spark/blob/v2.4.0/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L81
>
> I guess Spark developers had a good reason to fix it, but is it possible to make configurable the prefix of the above uniqueGroupId ("spark-kafka-source")?
>
> The rational is that spark users are not forced to use the same certificate on group-ids of the form (spark-kafka-source-*).
>
> DoD:
> * Allow spark SS users to define the group.id prefix as input parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org