You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Maxim Gekk <ma...@databricks.com> on 2022/01/12 18:39:30 UTC
Migration onto error classes and their testing
Hi All,
As you probably know, we introduced error classes (
https://github.com/apache/spark/pull/32850) as a part of the error messages
framework (https://issues.apache.org/jira/browse/SPARK-33539). I would like
to propose to migrate all exceptions from QueryExecutionErrors,
QueryCompilationErrors and QueryParsingErrors on the error classes using
instances of SparkThrowable, and carefully test every error class by
writing tests in dedicated test suites:
- QueryExecutionErrorsSuite for the errors that are occurred during query
execution
- QueryCompilationErrorsSuite ... query compilation or eagerly executing
commands
- QueryParsingErrorsSuite ... parsing errors
Here is an example https://github.com/apache/spark/pull/35157 of how an
existing Java exception can be replaced, and testing of related
error classes. If there will be no objections to the approach, I would
create an umbrella JIRA for this activity, and sub-tasks per every error
class (or a group of related classes). At the end, we should migrate all
exceptions from the files Query.*Errors.scala and cover all error classes
from the error-classes.json file by tests. WDYT, any thoughts are welcome.
Yours faithfully,
Max Gekk
Re: Migration onto error classes and their testing
Posted by Maxim Gekk <ma...@databricks.com>.
Hi All,
I have created the umbrella JIRA
https://issues.apache.org/jira/browse/SPARK-37935, and a few sub-tasks. If
you would like to contribute, please leave a comment in a sub-task that you
are working on it.
Yours faithfully,
Max Gekk
On Wed, Jan 12, 2022 at 9:39 PM Maxim Gekk <ma...@databricks.com>
wrote:
> Hi All,
>
> As you probably know, we introduced error classes (
> https://github.com/apache/spark/pull/32850) as a part of the error
> messages framework (https://issues.apache.org/jira/browse/SPARK-33539). I
> would like to propose to migrate all exceptions from QueryExecutionErrors,
> QueryCompilationErrors and QueryParsingErrors on the error classes using
> instances of SparkThrowable, and carefully test every error class by
> writing tests in dedicated test suites:
> - QueryExecutionErrorsSuite for the errors that are occurred during query
> execution
> - QueryCompilationErrorsSuite ... query compilation or eagerly executing
> commands
> - QueryParsingErrorsSuite ... parsing errors
>
> Here is an example https://github.com/apache/spark/pull/35157 of how an
> existing Java exception can be replaced, and testing of related
> error classes. If there will be no objections to the approach, I would
> create an umbrella JIRA for this activity, and sub-tasks per every error
> class (or a group of related classes). At the end, we should migrate all
> exceptions from the files Query.*Errors.scala and cover all error classes
> from the error-classes.json file by tests. WDYT, any thoughts are welcome.
>
> Yours faithfully,
> Max Gekk
>