[jira] [Created] (SPARK-26121) [Structured Streaming] Allow users to define prefix of Kafka's consumer group (group.id)

["Anastasios Zouzias (JIRA)" <ji...@apache.org>]

Anastasios Zouzias created SPARK-26121:
------------------------------------------

             Summary: [Structured Streaming] Allow users to define prefix of Kafka's consumer group (group.id)
                 Key: SPARK-26121
                 URL: https://issues.apache.org/jira/browse/SPARK-26121
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 2.4.0
            Reporter: Anastasios Zouzias


I run in the following situation with Spark Structure Streaming (SS) using Kafka.
 
In a project that I work on, there is already a secured Kafka setup where ops can issue an SSL certificate per "[group.id|http://group.id/]", which should be predefined (or its prefix to be predefined).
 
On the other hand, Spark SS fixes the [group.id|http://group.id/] to 
 
val uniqueGroupId = s"spark-kafka-source-${UUID.randomUUID}-${metadataPath.hashCode}"
 
see, i.e.,
 
[https://github.com/apache/spark/blob/v2.4.0/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L124]
 
https://github.com/apache/spark/blob/v2.4.0/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L81
 
I guess Spark developers had a good reason to fix it, but is it possible to make configurable the prefix of the above uniqueGroupId ("spark-kafka-source")?
 
The rational is that spark users are not forced to use the same certificate on group-ids of the form (spark-kafka-source-*).
 
DoD:
* Allow spark SS users to define the group.id prefix as input parameter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org