[jira] [Assigned] (SPARK-26121) [Structured Streaming] Allow users to define prefix of Kafka's consumer group (group.id)

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-26121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-26121:
------------------------------------

    Assignee:     (was: Apache Spark)

> [Structured Streaming] Allow users to define prefix of Kafka's consumer group (group.id)
> ----------------------------------------------------------------------------------------
>
>                 Key: SPARK-26121
>                 URL: https://issues.apache.org/jira/browse/SPARK-26121
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 2.4.0
>            Reporter: Anastasios Zouzias
>            Priority: Minor
>
> I run in the following situation with Spark Structure Streaming (SS) using Kafka.
>  
> In a project that I work on, there is already a secured Kafka setup where ops can issue an SSL certificate per "[group.id|http://group.id/]", which should be predefined (or its prefix to be predefined).
>  
> On the other hand, Spark SS fixes the [group.id|http://group.id/] to 
>  
> val uniqueGroupId = s"spark-kafka-source-${UUID.randomUUID}-${metadataPath.hashCode}"
>  
> see, i.e.,
>  
> [https://github.com/apache/spark/blob/v2.4.0/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L124]
>  
> https://github.com/apache/spark/blob/v2.4.0/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L81
>  
> I guess Spark developers had a good reason to fix it, but is it possible to make configurable the prefix of the above uniqueGroupId ("spark-kafka-source")?
>  
> The rational is that spark users are not forced to use the same certificate on group-ids of the form (spark-kafka-source-*).
>  
> DoD:
> * Allow spark SS users to define the group.id prefix as input parameter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org