[spark] branch branch-3.4 updated: [SPARK-43067][SS] Correct the location of error class resource file in Kafka connector

[ka...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

kabhwan pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.4 by this push:
     new 66b11477201 [SPARK-43067][SS] Correct the location of error class resource file in Kafka connector
66b11477201 is described below

commit 66b11477201f201b3826267854416e72a336445a
Author: Jungtaek Lim <ka...@gmail.com>
AuthorDate: Sun Apr 9 06:38:46 2023 +0900

    [SPARK-43067][SS] Correct the location of error class resource file in Kafka connector
    
    ### What changes were proposed in this pull request?
    
    This PR moves the error class resource file in Kafka connector from test to src, so that error class works without test artifacts.
    
    ### Why are the changes needed?
    
    Refer to the `How was this patch tested?`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, but the possibility of encountering this is small enough.
    
    ### How was this patch tested?
    
    Ran spark-shell with Kafka connector artifacts (without test artifacts) and triggered KafkaExceptions to confirm that exception is properly raised.
    
    ```
    scala> import org.apache.spark.sql.kafka010.KafkaExceptions
    import org.apache.spark.sql.kafka010.KafkaExceptions
    
    scala> import org.apache.kafka.common.TopicPartition
    import org.apache.kafka.common.TopicPartition
    
    scala> KafkaExceptions.mismatchedTopicPartitionsBetweenEndOffsetAndPrefetched(Set[TopicPartition](), Set[TopicPartition]())
    res1: org.apache.spark.SparkException =
    org.apache.spark.SparkException: Kafka data source in Trigger.AvailableNow should provide the same topic partitions in pre-fetched offset to end offset for each microbatch. The error could be transient - restart your query, and report if you still see the same issue.
    topic-partitions for pre-fetched offset: Set(), topic-partitions for end offset: Set().
    ```
    
    Without the fix, triggering KafkaExceptions failed to load error class resource file and led unexpected exception.
    
    ```
    scala> KafkaExceptions.mismatchedTopicPartitionsBetweenEndOffsetAndPrefetched(Set[TopicPartition](), Set[TopicPartition]())
    java.lang.IllegalArgumentException: argument "src" is null
      at com.fasterxml.jackson.databind.ObjectMapper._assertNotNull(ObjectMapper.java:4885)
      at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3618)
      at org.apache.spark.ErrorClassesJsonReader$.org$apache$spark$ErrorClassesJsonReader$$readAsMap(ErrorClassesJSONReader.scala:95)
      at org.apache.spark.ErrorClassesJsonReader.$anonfun$errorInfoMap$1(ErrorClassesJSONReader.scala:44)
      at scala.collection.immutable.List.map(List.scala:293)
      at org.apache.spark.ErrorClassesJsonReader.<init>(ErrorClassesJSONReader.scala:44)
      at org.apache.spark.sql.kafka010.KafkaExceptions$.<init>(KafkaExceptions.scala:27)
      at org.apache.spark.sql.kafka010.KafkaExceptions$.<clinit>(KafkaExceptions.scala)
      ... 47 elided
    ```
    
    Closes #40705 from HeartSaVioR/SPARK-43067.
    
    Authored-by: Jungtaek Lim <ka...@gmail.com>
    Signed-off-by: Jungtaek Lim <ka...@gmail.com>
    (cherry picked from commit 7434702a120360593a10ef6b3eb9ba0fd7e37de1)
    Signed-off-by: Jungtaek Lim <ka...@gmail.com>
---
 .../src/{test => main}/resources/error/kafka-error-classes.json           | 0
 1 file changed, 0 insertions(+), 0 deletions(-)

diff --git a/connector/kafka-0-10-sql/src/test/resources/error/kafka-error-classes.json b/connector/kafka-0-10-sql/src/main/resources/error/kafka-error-classes.json
similarity index 100%
rename from connector/kafka-0-10-sql/src/test/resources/error/kafka-error-classes.json
rename to connector/kafka-0-10-sql/src/main/resources/error/kafka-error-classes.json


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org