[jira] [Commented] (SPARK-39412) IllegalStateException from connector does not work well with error class framework

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-39412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17551463#comment-17551463 ] 

Apache Spark commented on SPARK-39412:
--------------------------------------

User 'MaxGekk' has created a pull request for this issue:
https://github.com/apache/spark/pull/36804

> IllegalStateException from connector does not work well with error class framework
> ----------------------------------------------------------------------------------
>
>                 Key: SPARK-39412
>                 URL: https://issues.apache.org/jira/browse/SPARK-39412
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 3.3.0
>            Reporter: Jungtaek Lim
>            Priority: Blocker
>         Attachments: kafka-dataloss-error-msg-in-spark-3-2.log, kafka-dataloss-error-msg-in-spark-3-3-or-master.log
>
>
> With SPARK-39346, Spark SQL binds several exceptions to the internal error, and produces different guidance on dealing with the exception. This assumes these exceptions are only used for noticing internal bugs.
> This applies to "connectors" as well, and introduces side-effect on the error log. For Kafka data source, it is a breaking and unacceptable change, because there is an important use case Kafka data source determines a case of "dataloss", and throws IllegalStateException with instruction message on workaround.
> I mentioned this as "important" use case, because it can even happen with some valid scenarios - streaming query has some maintenance period and Kafka's retention on topic removes some records in the meanwhile.
> Two problems arise:
> 1) This does not mean Spark has a bug and end users have to report, hence the guidance message on internal error is misleading.
> 2) Most importantly, instruction message is shown after a long stack trace. With the modification of existing test suite, I see the message being appeared in "line 90" of the error log.
> We should roll the right error message back, at least for Kafka's case.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org