[jira] [Created] (SPARK-26360) Avoid extra validateQuery call in createStreamingWriteSupport

["Wu Wenjie (JIRA)" <ji...@apache.org>]

Wu Wenjie created SPARK-26360:
---------------------------------

             Summary: Avoid extra validateQuery call in createStreamingWriteSupport
                 Key: SPARK-26360
                 URL: https://issues.apache.org/jira/browse/SPARK-26360
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 2.4.0, 2.3.2
            Reporter: Wu Wenjie


When I'm reading structured streaming source code, I find there is a extra KafkaWriter.validateQuery() function call in createStreamingWriteSupport func in class 

KafkaSourceProvider.

{code:scala}
// KafkaSourceProvider.scala
  override def createStreamingWriteSupport(
      queryId: String,
      schema: StructType,
      mode: OutputMode,
      options: DataSourceOptions): StreamingWriteSupport = {
   .....
    // validate once here
    KafkaWriter.validateQuery(schema.toAttributes, producerParams, topic)

    // validate twice here
    new KafkaStreamingWriteSupport(topic, producerParams, schema)
  }

// KafkaStreamingWriteSupport.scala
class KafkaStreamingWriteSupport(
    topic: Option[String],
    producerParams: ju.Map[String, Object],
    schema: StructType)
  extends StreamingWriteSupport {

  validateQuery(schema.toAttributes, producerParams, topic)
  ....
}
{code}

 

I think we just need to remove it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org