You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/24 07:18:29 UTC
[GitHub] [spark] Peng-Lei opened a new pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Peng-Lei opened a new pull request #34093:
URL: https://github.com/apache/spark/pull/34093
### What changes were proposed in this pull request?
Refactor fifth set of 20 query execution errors to use error classes. as follows:
`
createStreamingSourceNotSpecifySchemaError
streamedOperatorUnsupportedByDataSourceError
multiplePathsSpecifiedError
failedToFindDataSourceError
removedClassInSpark2Error
incompatibleDataSourceRegisterError
unrecognizedFileFormatError
sparkUpgradeInReadingDatesError
sparkUpgradeInWritingDatesError
buildReaderUnsupportedForFileFormatError
jobAbortedError
taskFailedWhileWritingRowsError
readCurrentFileNotFoundError
unsupportedSaveModeError
cannotClearOutputDirectoryError
cannotClearPartitionDirectoryError
failedToCastValueToDataTypeForPartitionColumnError
endOfStreamError
fallbackV1RelationReportsInconsistentSchemaError
cannotDropNonemptyNamespaceError
`
### Why are the changes needed?
[SPARK-36294](https://issues.apache.org/jira/browse/SPARK-36294)
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existed UT Testcase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-942962575
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48712/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-945643702
**[Test build #144363 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144363/testReport)** for PR 34093 at commit [`e7150d8`](https://github.com/apache/spark/commit/e7150d824f55c6d97bf3b109607959c8df911c32).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-943086979
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144234/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] karenfeng commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
karenfeng commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r718729445
##########
File path: core/src/main/scala/org/apache/spark/SparkException.scala
##########
@@ -72,9 +72,47 @@ private[spark] case class ExecutorDeadException(message: String)
/**
* Exception thrown when Spark returns different result after upgrading to a new version.
*/
-private[spark] class SparkUpgradeException(version: String, message: String, cause: Throwable)
+private[spark] class SparkUpgradeException(
+ version: String,
+ message: String,
+ cause: Throwable,
+ errorClass: Option[String],
+ messageParameters: Array[String])
extends RuntimeException("You may get a different result due to the upgrading of Spark" +
- s" $version: $message", cause)
+ s" $version: $message", cause) with SparkThrowable {
+
+ def this(version: String, message: String, cause: Throwable) =
+ this(
+ version = version,
+ message = message,
+ cause = cause,
+ errorClass = None,
+ messageParameters = Array.empty
+ )
+
+ def this(errorClass: String, messageParameters: Array[String], cause: Throwable) =
Review comment:
I don't know if we want to have this constructor - it feels like the version should always be provided.
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +53,32 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:", "Schema of v2 scan: %s", "Schema of v1 relation: %s" ],
+ "sqlState" : "22023"
+ },
+ "FAILED_FIND_DATA_SOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
+ },
+ "FAILED_FORMAT_DATETIME_IN_NEW_FORMATTER" : {
Review comment:
I think we can unify these names. How about: `FAILED_FORMAT_DATETIME_IN_SPARK3`?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +57,31 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:\n Schema of v2 scan: %s\nSchema of v1 relation: %s" ]
+ },
+ "FAILED_FIND_DATASOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
Review comment:
Bump!
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +53,32 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:", "Schema of v2 scan: %s", "Schema of v1 relation: %s" ],
+ "sqlState" : "22023"
+ },
+ "FAILED_FIND_DATA_SOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
+ },
+ "FAILED_FORMAT_DATETIME_IN_NEW_FORMATTER" : {
+ "message" : [ "Fail to format it to '%s' in the new formatter. You can set %s to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string." ]
+ },
+ "FAILED_PARSE_DATETIME_IN_NEW_PARSER" : {
Review comment:
How about: `FAILED_PARSE_DATETIME_IN_SPARK3`?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -153,13 +215,24 @@
"message" : [ "Unsupported literal type %s %s" ],
"sqlState" : "0A000"
},
+ "UNSUPPORTED_SAVE_MODE" : {
+ "message" : [ "unsupported save mode %s" ],
+ "sqlState" : "0A000"
+ },
"UNSUPPORTED_SIMPLE_STRING_WITH_NODE_ID" : {
"message" : [ "%s does not implement simpleStringWithNodeId" ]
},
+ "UNSUPPORTED_STREAMED_OPERATOR_BY_DATA_SOURCE" : {
+ "message" : [ "Data source %s does not support streamed %s" ],
+ "sqlState" : "0A000"
+ },
"UNSUPPORTED_TRANSACTION_BY_JDBC_SERVER" : {
"message" : [ "The target JDBC server does not support transaction and can only support ALTER TABLE with a single action." ],
"sqlState" : "0A000"
},
+ "WRITING_AMBIGUOUS_DATES" : {
Review comment:
For these upgrade class names, should we always include the Spark version name? Maybe `WRITING_AMBIGUOUS_NAMES_IN_SPARK3`
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +53,32 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:", "Schema of v2 scan: %s", "Schema of v1 relation: %s" ],
+ "sqlState" : "22023"
+ },
+ "FAILED_FIND_DATA_SOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
+ },
+ "FAILED_FORMAT_DATETIME_IN_NEW_FORMATTER" : {
+ "message" : [ "Fail to format it to '%s' in the new formatter. You can set %s to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string." ]
+ },
+ "FAILED_PARSE_DATETIME_IN_NEW_PARSER" : {
+ "message" : [ "Fail to parse '%s' in the new parser. You can set %s to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string." ]
+ },
+ "FAILED_RECOGNIZE_PATTERN_AFTER_UPGRADE" : {
Review comment:
How about: `FAILED_RECOGNIZE_DATETIME_PATTERN_IN_SPARK3`?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -122,6 +166,9 @@
"message" : [ "Invalid pivot value '%s': value data type %s does not match pivot column data type %s" ],
"sqlState" : "42000"
},
+ "READING_AMBIGUOUS_DATES" : {
Review comment:
For these upgrade class names, should we always include the Spark version name? Maybe `READING_AMBIGUOUS_NAMES_IN_SPARK3`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-946443515
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144390/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-946443515
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144390/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-946381470
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48864/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-946381470
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48864/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-945581053
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48838/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] karenfeng commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
karenfeng commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r726629302
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -435,131 +435,129 @@ object QueryExecutionErrors {
}
def createStreamingSourceNotSpecifySchemaError(): Throwable = {
- new IllegalArgumentException(
- s"""
- |Schema must be specified when creating a streaming source DataFrame. If some
- |files already exist in the directory, then depending on the file format you
- |may be able to create a static DataFrame on that directory with
- |'spark.read.load(directory)' and infer schema from it.
- """.stripMargin)
+ new SparkIllegalArgumentException(
+ errorClass = "MISSING_STREAMING_SOURCE_SCHEMA",
+ messageParameters = Array.empty)
}
def streamedOperatorUnsupportedByDataSourceError(
className: String, operator: String): Throwable = {
- new UnsupportedOperationException(
- s"Data source $className does not support streamed $operator")
+ new SparkUnsupportedOperationException(
+ errorClass = "UNSUPPORTED_STREAMED_OPERATOR_BY_DATA_SOURCE",
+ messageParameters = Array(className, operator))
}
def multiplePathsSpecifiedError(allPaths: Seq[String]): Throwable = {
- new IllegalArgumentException("Expected exactly one path to be specified, but " +
- s"got: ${allPaths.mkString(", ")}")
+ new SparkIllegalArgumentException(
+ errorClass = "SPECIFIED_MULTIPLE_PATHS",
+ messageParameters = Array(allPaths.mkString(", ")))
}
def failedToFindDataSourceError(provider: String, error: Throwable): Throwable = {
- new ClassNotFoundException(
- s"""
- |Failed to find data source: $provider. Please find packages at
- |http://spark.apache.org/third-party-projects.html
- """.stripMargin, error)
+ new SparkClassNotFoundException(
+ errorClass = "FAILED_FIND_DATA_SOURCE",
+ messageParameters = Array(provider), error)
}
def removedClassInSpark2Error(className: String, e: Throwable): Throwable = {
- new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
- "Please check if your library is compatible with Spark 2.0", e)
+ new SparkClassNotFoundException(
+ errorClass = "CANNOT_FIND_CLASS_IN_SPARK2",
+ messageParameters = Array(className), e)
}
def incompatibleDataSourceRegisterError(e: Throwable): Throwable = {
- new SparkClassNotFoundException("INCOMPATIBLE_DATASOURCE_REGISTER", Array(e.getMessage), e)
+ new SparkClassNotFoundException(
+ errorClass = "INCOMPATIBLE_DATASOURCE_REGISTER",
+ messageParameters = Array(e.getMessage), e)
}
def unrecognizedFileFormatError(format: String): Throwable = {
- new IllegalStateException(s"unrecognized format $format")
+ new SparkIllegalStateException(
+ errorClass = "UNRECOGNIZED_FORMAT",
+ messageParameters = Array(format))
}
def sparkUpgradeInReadingDatesError(
format: String, config: String, option: String): SparkUpgradeException = {
- new SparkUpgradeException("3.0",
- s"""
- |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
- |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
- |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
- |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
- |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
- |w.r.t. the calendar difference during reading. To read the datetime values as it is,
- |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
- """.stripMargin, null)
+ new SparkUpgradeException(
+ version = "3.0",
+ errorClass = "READING_AMBIGUOUS_DATES_AFTER_UPGRADE",
+ messageParameters = Array(format, config, option, config, option), null)
}
def sparkUpgradeInWritingDatesError(format: String, config: String): SparkUpgradeException = {
- new SparkUpgradeException("3.0",
- s"""
- |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
- |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
- |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
- |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
- |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
- |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
- |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
- |systems that use Proleptic Gregorian calendar.
- """.stripMargin, null)
+ new SparkUpgradeException(
+ version = "3.0",
+ errorClass = "WRITING_AMBIGUOUS_DATES_AFTER_UPGRADE",
+ messageParameters = Array(format, config, config), null)
}
def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
- new UnsupportedOperationException(s"buildReader is not supported for $format")
+ new SparkUnsupportedOperationException(
+ errorClass = "UNSUPPORTED_BUILD_READER_FOR_FILE_FORMAT",
+ messageParameters = Array(format))
}
def jobAbortedError(cause: Throwable): Throwable = {
- new SparkException("Job aborted.", cause)
+ new SparkException(
+ errorClass = "JOB_ABORTED",
+ messageParameters = Array.empty, cause)
}
def taskFailedWhileWritingRowsError(cause: Throwable): Throwable = {
- new SparkException("Task failed while writing rows.", cause)
+ new SparkException(
+ errorClass = "TASK_FAILED_WRITING_ROWS",
+ messageParameters = Array.empty, cause)
}
def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
- new FileNotFoundException(
- s"""
- |${e.getMessage}\n
- |It is possible the underlying files have been updated. You can explicitly invalidate
- |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
- |recreating the Dataset/DataFrame involved.
- """.stripMargin)
+ new SparkFileNotFoundException(
+ errorClass = "CANNOT_READ_CURRENT_FILE",
+ messageParameters = Array(e.getMessage))
}
def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
- new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ new SparkIllegalStateException(
+ errorClass = "UNSUPPORTED_SAVE_MODE",
+ messageParameters = Array(saveMode + " (" + pathExists.toString + ")"))
}
def cannotClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
- new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ new SparkIOException(
+ errorClass = "CANNOT_CLEAR_DIRECTORY",
+ messageParameters = Array("output", staticPrefixPath.toString))
}
def cannotClearPartitionDirectoryError(path: Path): Throwable = {
- new IOException(s"Unable to clear partition directory $path prior to writing to it")
+ new SparkIOException(
+ errorClass = "CANNOT_CLEAR_DIRECTORY",
+ messageParameters = Array("partition", path.toString))
}
def failedToCastValueToDataTypeForPartitionColumnError(
value: String, dataType: DataType, columnName: String): Throwable = {
- new RuntimeException(s"Failed to cast value `$value` to " +
- s"`$dataType` for partition column `$columnName`")
+ new SparkRuntimeException(
+ errorClass = "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN",
+ messageParameters = Array(value, dataType.toString, columnName))
}
def endOfStreamError(): Throwable = {
- new NoSuchElementException("End of stream")
+ new SparkNoSuchElementException(
+ errorClass = "END_OF_STREAM",
+ messageParameters = Array.empty)
}
def fallbackV1RelationReportsInconsistentSchemaError(
v2Schema: StructType, v1Schema: StructType): Throwable = {
- new IllegalArgumentException(
- "The fallback v1 relation reports inconsistent schema:\n" +
- "Schema of v2 scan: " + v2Schema + "\n" +
- "Schema of v1 relation: " + v1Schema)
+ new SparkIllegalArgumentException(
+ errorClass = "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA",
Review comment:
This actually looks like it might be an internal error - @cloud-fan, can you confirm? If so, please use `INTERNAL_ERROR` instead.
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +57,31 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:\n Schema of v2 scan: %s\nSchema of v1 relation: %s" ]
+ },
+ "FAILED_FIND_DATASOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
Review comment:
Bump again
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
+ },
"CANNOT_EVALUATE_EXPRESSION" : {
"message" : [ "Cannot evaluate expression: %s" ]
},
+ "CANNOT_FIND_CLASS_IN_SPARK2" : {
+ "message" : [ "%s was removed in Spark 2.0. Please check if your library is compatible with Spark 2.0" ]
+ },
"CANNOT_GENERATE_CODE_FOR_EXPRESSION" : {
"message" : [ "Cannot generate code for expression: %s" ]
},
"CANNOT_PARSE_DECIMAL" : {
"message" : [ "Cannot parse decimal" ],
"sqlState" : "42000"
},
+ "CANNOT_READ_CURRENT_FILE" : {
+ "message" : [ "%s \n It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved." ]
+ },
"CANNOT_TERMINATE_GENERATOR" : {
"message" : [ "Cannot terminate expression: %s" ]
},
+ "CANNOT_UPGRADE_IN_READING_DATES" : {
Review comment:
I believe this class is no longer used, can you remove it?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
+ },
"CANNOT_EVALUATE_EXPRESSION" : {
"message" : [ "Cannot evaluate expression: %s" ]
},
+ "CANNOT_FIND_CLASS_IN_SPARK2" : {
+ "message" : [ "%s was removed in Spark 2.0. Please check if your library is compatible with Spark 2.0" ]
+ },
"CANNOT_GENERATE_CODE_FOR_EXPRESSION" : {
"message" : [ "Cannot generate code for expression: %s" ]
},
"CANNOT_PARSE_DECIMAL" : {
"message" : [ "Cannot parse decimal" ],
"sqlState" : "42000"
},
+ "CANNOT_READ_CURRENT_FILE" : {
+ "message" : [ "%s \n It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved." ]
+ },
"CANNOT_TERMINATE_GENERATOR" : {
"message" : [ "Cannot terminate expression: %s" ]
},
+ "CANNOT_UPGRADE_IN_READING_DATES" : {
+ "message" : [ "You may get a different result due to the upgrading of Spark %s reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from %s files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set the SQL config '%s' or the datasource option '%s' to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during reading. To read the datetime values as it is, set the SQL config '%s' or the datasource option '%s' to 'CORRECTED'." ]
+ },
+ "CANNOT_UPGRADE_IN_WRITING_DATES" : {
Review comment:
I believe this class is no longer used, can you remove it?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -121,13 +174,28 @@
"message" : [ "The second argument of '%s' function needs to be an integer." ],
"sqlState" : "22023"
},
+ "SPECIFIED_MULTIPLE_PATHS" : {
+ "message" : [ "Expected exactly one path to be specified, but got: %s" ],
+ "sqlState" : "22023"
+ },
+ "TASK_FAILED_WRITING_ROWS" : {
Review comment:
This grammar is a bit strange; maybe `FAILED_TASK_WHILE_WRITING_ROWS`?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +53,32 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:", "Schema of v2 scan: %s", "Schema of v1 relation: %s" ],
+ "sqlState" : "22023"
+ },
+ "FAILED_FIND_DATA_SOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
+ },
+ "FAILED_FORMAT_DATETIME_IN_NEW_FORMATTER" : {
+ "message" : [ "Fail to format it to '%s' in the new formatter. You can set %s to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string." ]
+ },
+ "FAILED_PARSE_DATETIME_IN_NEW_PARSER" : {
+ "message" : [ "Fail to parse '%s' in the new parser. You can set %s to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string." ]
+ },
+ "FAILED_RECOGNIZE_PATTERN_AFTER_UPGRADE" : {
Review comment:
Bump!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-926416101
@karenfeng @HyukjinKwon Could you take a look ? Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-946341629
**[Test build #144390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144390/testReport)** for PR 34093 at commit [`4088c77`](https://github.com/apache/spark/commit/4088c77dfe07b3b9a5449a95dd1d89431b6b2766).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r728601622
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -121,13 +174,28 @@
"message" : [ "The second argument of '%s' function needs to be an integer." ],
"sqlState" : "22023"
},
+ "SPECIFIED_MULTIPLE_PATHS" : {
+ "message" : [ "Expected exactly one path to be specified, but got: %s" ],
+ "sqlState" : "22023"
+ },
+ "TASK_FAILED_WRITING_ROWS" : {
Review comment:
ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-946423397
**[Test build #144390 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144390/testReport)** for PR 34093 at commit [`4088c77`](https://github.com/apache/spark/commit/4088c77dfe07b3b9a5449a95dd1d89431b6b2766).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-942936157
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48712/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r730636751
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -30,16 +44,43 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:", "Schema of v2 scan: %s", "Schema of v1 relation: %s" ],
+ "sqlState" : "22023"
Review comment:
ok, I also think it is a internal error. I will update it. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-945630946
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48838/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r715385719
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -110,6 +153,10 @@
"message" : [ "Unknown static partition column: %s" ],
"sqlState" : "42000"
},
+ "MISSING_STREAMING_SOURCE_SCHEMA" : {
+ "message" : [ "Schema must be specified when creating a streaming source DataFrame. If some files already exist in the directory, then depending on the file format you may be able to create a static DataFrame on that directory with 'spark.read.load(directory)' and infer schema from it." ],
+ "sqlState" : "3F000"
Review comment:
invalid schema name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-926403952
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r715386251
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -134,13 +181,28 @@
"message" : [ "The second argument of '%s' function needs to be an integer." ],
"sqlState" : "22023"
},
+ "SPECIFIED_MULTIPLE_PATHS" : {
+ "message" : [ "Expected exactly one path to be specified, but got: %s" ],
+ "sqlState" : "22023"
+ },
+ "TASK_FAILED_WRITING_ROWS" : {
+ "message" : [ "Task failed while writing rows." ]
+ },
"UNABLE_TO_ACQUIRE_MEMORY" : {
"message" : [ "Unable to acquire %s bytes of memory, got %s" ]
},
+ "UNRECOGNIZED_FORMAT" : {
+ "message" : [ "unrecognized format %s" ],
+ "sqlState" : "22023"
+ },
"UNRECOGNIZED_SQL_TYPE" : {
"message" : [ "Unrecognized SQL type %s" ],
"sqlState" : "42000"
},
+ "UNSUPPORTED_BUILD_READER_FOR_FILE_FORMAT" : {
+ "message" : [ "buildReader is not supported for %s" ],
+ "sqlState" : "0A000"
Review comment:
feature not supported
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-927253706
> Thanks for working on this
It's my pleasure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei closed pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei closed pull request #34093:
URL: https://github.com/apache/spark/pull/34093
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-946341629
**[Test build #144390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144390/testReport)** for PR 34093 at commit [`4088c77`](https://github.com/apache/spark/commit/4088c77dfe07b3b9a5449a95dd1d89431b6b2766).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] karenfeng commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
karenfeng commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r726628246
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +53,32 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:", "Schema of v2 scan: %s", "Schema of v1 relation: %s" ],
+ "sqlState" : "22023"
+ },
+ "FAILED_FIND_DATA_SOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
+ },
+ "FAILED_FORMAT_DATETIME_IN_NEW_FORMATTER" : {
+ "message" : [ "Fail to format it to '%s' in the new formatter. You can set %s to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string." ]
+ },
+ "FAILED_PARSE_DATETIME_IN_NEW_PARSER" : {
+ "message" : [ "Fail to parse '%s' in the new parser. You can set %s to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string." ]
+ },
+ "FAILED_RECOGNIZE_PATTERN_AFTER_UPGRADE" : {
Review comment:
Bump!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-930348735
@karenfeng Done. Cloud you take a look? Thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-942962575
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48712/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r718095641
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -110,6 +153,10 @@
"message" : [ "Unknown static partition column: %s" ],
"sqlState" : "42000"
},
+ "MISSING_STREAMING_SOURCE_SCHEMA" : {
+ "message" : [ "Schema must be specified when creating a streaming source DataFrame. If some files already exist in the directory, then depending on the file format you may be able to create a static DataFrame on that directory with 'spark.read.load(directory)' and infer schema from it." ],
+ "sqlState" : "3F000"
Review comment:
Agree with you. Thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r718104419
##########
File path: core/src/main/scala/org/apache/spark/SparkException.scala
##########
@@ -72,9 +72,11 @@ private[spark] case class ExecutorDeadException(message: String)
/**
* Exception thrown when Spark returns different result after upgrading to a new version.
*/
-private[spark] class SparkUpgradeException(version: String, message: String, cause: Throwable)
- extends RuntimeException("You may get a different result due to the upgrading of Spark" +
- s" $version: $message", cause)
+private[spark] class SparkUpgradeException(
Review comment:
I just think it's private, but it is more safe that use overload instead of modify it. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-945538917
@karenfeng I update again. Cloud you take a look? Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei closed pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei closed pull request #34093:
URL: https://github.com/apache/spark/pull/34093
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r718096765
##########
File path: R/pkg/tests/fulltests/test_sparkSQL.R
##########
@@ -3873,7 +3873,8 @@ test_that("Call DataFrameWriter.save() API in Java without path and check argume
# It makes sure that we can omit path argument in write.df API and then it calls
# DataFrameWriter.save() without path.
expect_error(write.df(df, source = "csv"),
- "Error in save : illegal argument - Expected exactly one path to be specified")
+ paste("Error in save : org.apache.spark.SparkIllegalArgumentException:",
Review comment:
Thank you for the reminder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-942962546
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48712/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r730690648
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -435,131 +435,129 @@ object QueryExecutionErrors {
}
def createStreamingSourceNotSpecifySchemaError(): Throwable = {
- new IllegalArgumentException(
- s"""
- |Schema must be specified when creating a streaming source DataFrame. If some
- |files already exist in the directory, then depending on the file format you
- |may be able to create a static DataFrame on that directory with
- |'spark.read.load(directory)' and infer schema from it.
- """.stripMargin)
+ new SparkIllegalArgumentException(
+ errorClass = "MISSING_STREAMING_SOURCE_SCHEMA",
+ messageParameters = Array.empty)
}
def streamedOperatorUnsupportedByDataSourceError(
className: String, operator: String): Throwable = {
- new UnsupportedOperationException(
- s"Data source $className does not support streamed $operator")
+ new SparkUnsupportedOperationException(
+ errorClass = "UNSUPPORTED_STREAMED_OPERATOR_BY_DATA_SOURCE",
+ messageParameters = Array(className, operator))
}
def multiplePathsSpecifiedError(allPaths: Seq[String]): Throwable = {
- new IllegalArgumentException("Expected exactly one path to be specified, but " +
- s"got: ${allPaths.mkString(", ")}")
+ new SparkIllegalArgumentException(
+ errorClass = "SPECIFIED_MULTIPLE_PATHS",
+ messageParameters = Array(allPaths.mkString(", ")))
}
def failedToFindDataSourceError(provider: String, error: Throwable): Throwable = {
- new ClassNotFoundException(
- s"""
- |Failed to find data source: $provider. Please find packages at
- |http://spark.apache.org/third-party-projects.html
- """.stripMargin, error)
+ new SparkClassNotFoundException(
+ errorClass = "FAILED_FIND_DATA_SOURCE",
+ messageParameters = Array(provider), error)
}
def removedClassInSpark2Error(className: String, e: Throwable): Throwable = {
- new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
- "Please check if your library is compatible with Spark 2.0", e)
+ new SparkClassNotFoundException(
+ errorClass = "CANNOT_FIND_CLASS_IN_SPARK2",
+ messageParameters = Array(className), e)
}
def incompatibleDataSourceRegisterError(e: Throwable): Throwable = {
- new SparkClassNotFoundException("INCOMPATIBLE_DATASOURCE_REGISTER", Array(e.getMessage), e)
+ new SparkClassNotFoundException(
+ errorClass = "INCOMPATIBLE_DATASOURCE_REGISTER",
+ messageParameters = Array(e.getMessage), e)
}
def unrecognizedFileFormatError(format: String): Throwable = {
- new IllegalStateException(s"unrecognized format $format")
+ new SparkIllegalStateException(
+ errorClass = "UNRECOGNIZED_FORMAT",
+ messageParameters = Array(format))
}
def sparkUpgradeInReadingDatesError(
format: String, config: String, option: String): SparkUpgradeException = {
- new SparkUpgradeException("3.0",
- s"""
- |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
- |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
- |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
- |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
- |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
- |w.r.t. the calendar difference during reading. To read the datetime values as it is,
- |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
- """.stripMargin, null)
+ new SparkUpgradeException(
+ version = "3.0",
+ errorClass = "READING_AMBIGUOUS_DATES_AFTER_UPGRADE",
+ messageParameters = Array(format, config, option, config, option), null)
}
def sparkUpgradeInWritingDatesError(format: String, config: String): SparkUpgradeException = {
- new SparkUpgradeException("3.0",
- s"""
- |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
- |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
- |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
- |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
- |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
- |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
- |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
- |systems that use Proleptic Gregorian calendar.
- """.stripMargin, null)
+ new SparkUpgradeException(
+ version = "3.0",
+ errorClass = "WRITING_AMBIGUOUS_DATES_AFTER_UPGRADE",
+ messageParameters = Array(format, config, config), null)
}
def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
- new UnsupportedOperationException(s"buildReader is not supported for $format")
+ new SparkUnsupportedOperationException(
+ errorClass = "UNSUPPORTED_BUILD_READER_FOR_FILE_FORMAT",
+ messageParameters = Array(format))
}
def jobAbortedError(cause: Throwable): Throwable = {
- new SparkException("Job aborted.", cause)
+ new SparkException(
+ errorClass = "JOB_ABORTED",
+ messageParameters = Array.empty, cause)
}
def taskFailedWhileWritingRowsError(cause: Throwable): Throwable = {
- new SparkException("Task failed while writing rows.", cause)
+ new SparkException(
+ errorClass = "FAILED_TASK_WHILE_WRITING_ROWS",
+ messageParameters = Array.empty, cause)
}
def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
- new FileNotFoundException(
- s"""
- |${e.getMessage}\n
- |It is possible the underlying files have been updated. You can explicitly invalidate
- |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
- |recreating the Dataset/DataFrame involved.
- """.stripMargin)
+ new SparkFileNotFoundException(
+ errorClass = "CANNOT_READ_CURRENT_FILE",
+ messageParameters = Array(e.getMessage))
}
def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
- new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ new SparkIllegalStateException(
+ errorClass = "UNSUPPORTED_SAVE_MODE",
+ messageParameters = Array(saveMode + " (" + pathExists.toString + ")"))
Review comment:
I agree [#comments](https://github.com/apache/spark/pull/34093#discussion_r729995285).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-946358265
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48864/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-946373690
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48864/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-945644647
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144363/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-938528471
@karenfeng Sorry I'm late. A seven-day National Day holiday. I've updated the code based on your comments. If you're free, I'd like to ask you to look again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] karenfeng commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
karenfeng commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r718729445
##########
File path: core/src/main/scala/org/apache/spark/SparkException.scala
##########
@@ -72,9 +72,47 @@ private[spark] case class ExecutorDeadException(message: String)
/**
* Exception thrown when Spark returns different result after upgrading to a new version.
*/
-private[spark] class SparkUpgradeException(version: String, message: String, cause: Throwable)
+private[spark] class SparkUpgradeException(
+ version: String,
+ message: String,
+ cause: Throwable,
+ errorClass: Option[String],
+ messageParameters: Array[String])
extends RuntimeException("You may get a different result due to the upgrading of Spark" +
- s" $version: $message", cause)
+ s" $version: $message", cause) with SparkThrowable {
+
+ def this(version: String, message: String, cause: Throwable) =
+ this(
+ version = version,
+ message = message,
+ cause = cause,
+ errorClass = None,
+ messageParameters = Array.empty
+ )
+
+ def this(errorClass: String, messageParameters: Array[String], cause: Throwable) =
Review comment:
I don't know if we want to have this constructor - it feels like the version should always be provided.
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +53,32 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:", "Schema of v2 scan: %s", "Schema of v1 relation: %s" ],
+ "sqlState" : "22023"
+ },
+ "FAILED_FIND_DATA_SOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
+ },
+ "FAILED_FORMAT_DATETIME_IN_NEW_FORMATTER" : {
Review comment:
I think we can unify these names. How about: `FAILED_FORMAT_DATETIME_IN_SPARK3`?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +57,31 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:\n Schema of v2 scan: %s\nSchema of v1 relation: %s" ]
+ },
+ "FAILED_FIND_DATASOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
Review comment:
Bump!
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +53,32 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:", "Schema of v2 scan: %s", "Schema of v1 relation: %s" ],
+ "sqlState" : "22023"
+ },
+ "FAILED_FIND_DATA_SOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
+ },
+ "FAILED_FORMAT_DATETIME_IN_NEW_FORMATTER" : {
+ "message" : [ "Fail to format it to '%s' in the new formatter. You can set %s to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string." ]
+ },
+ "FAILED_PARSE_DATETIME_IN_NEW_PARSER" : {
Review comment:
How about: `FAILED_PARSE_DATETIME_IN_SPARK3`?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -153,13 +215,24 @@
"message" : [ "Unsupported literal type %s %s" ],
"sqlState" : "0A000"
},
+ "UNSUPPORTED_SAVE_MODE" : {
+ "message" : [ "unsupported save mode %s" ],
+ "sqlState" : "0A000"
+ },
"UNSUPPORTED_SIMPLE_STRING_WITH_NODE_ID" : {
"message" : [ "%s does not implement simpleStringWithNodeId" ]
},
+ "UNSUPPORTED_STREAMED_OPERATOR_BY_DATA_SOURCE" : {
+ "message" : [ "Data source %s does not support streamed %s" ],
+ "sqlState" : "0A000"
+ },
"UNSUPPORTED_TRANSACTION_BY_JDBC_SERVER" : {
"message" : [ "The target JDBC server does not support transaction and can only support ALTER TABLE with a single action." ],
"sqlState" : "0A000"
},
+ "WRITING_AMBIGUOUS_DATES" : {
Review comment:
For these upgrade class names, should we always include the Spark version name? Maybe `WRITING_AMBIGUOUS_NAMES_IN_SPARK3`
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +53,32 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:", "Schema of v2 scan: %s", "Schema of v1 relation: %s" ],
+ "sqlState" : "22023"
+ },
+ "FAILED_FIND_DATA_SOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
+ },
+ "FAILED_FORMAT_DATETIME_IN_NEW_FORMATTER" : {
+ "message" : [ "Fail to format it to '%s' in the new formatter. You can set %s to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string." ]
+ },
+ "FAILED_PARSE_DATETIME_IN_NEW_PARSER" : {
+ "message" : [ "Fail to parse '%s' in the new parser. You can set %s to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string." ]
+ },
+ "FAILED_RECOGNIZE_PATTERN_AFTER_UPGRADE" : {
Review comment:
How about: `FAILED_RECOGNIZE_DATETIME_PATTERN_IN_SPARK3`?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -122,6 +166,9 @@
"message" : [ "Invalid pivot value '%s': value data type %s does not match pivot column data type %s" ],
"sqlState" : "42000"
},
+ "READING_AMBIGUOUS_DATES" : {
Review comment:
For these upgrade class names, should we always include the Spark version name? Maybe `READING_AMBIGUOUS_NAMES_IN_SPARK3`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r715385928
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -134,13 +181,28 @@
"message" : [ "The second argument of '%s' function needs to be an integer." ],
"sqlState" : "22023"
},
+ "SPECIFIED_MULTIPLE_PATHS" : {
+ "message" : [ "Expected exactly one path to be specified, but got: %s" ],
+ "sqlState" : "22023"
Review comment:
invalid parameter value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-942931630
**[Test build #144234 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144234/testReport)** for PR 34093 at commit [`c8c16ad`](https://github.com/apache/spark/commit/c8c16ad50e01c6c5d247a0b6a1c15c07f9f92010).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-927304176
> Seems related test failure:
>
> ```
> `write.df(df, source = "csv")` threw an error with unexpected message.
> Expected match: "Error in save : org.apache.spark.SparkIllegalArgumentException: Expected exactly one path to be specified"
> Actual message: "Error in save : org.apache.spark.SparkIllegalArgumentException: Expected exactly one path to be specified, but got: \n\tat org.apache.spark.sql.errors.QueryExecutionErrors$.multiplePathsSpecifiedError(QueryExecutionErrors.scala:450)\n\tat org.apache.spark.sql.execution.datasources.DataSource.planForWritingFileFormat(DataSource.scala:464)\n\tat org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:558)\n\tat org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:382)\n\tat org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:355)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:247)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)
\n\tat org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:164)\n\tat org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:105)\n\tat org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:39)\n\tat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365
)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandle
rContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecu
tor.java:986)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.lang.Thread.run(Thread.java:748)\n\n"
> Backtrace:
> 1. testthat::expect_error(...) test_sparkSQL.R:3875:2
> 7. SparkR::write.df(df, source = "csv")
> 8. SparkR:::.local(df, path, ...)
> 9. SparkR:::handledCallJMethod(write, "save")
> 10. base::tryCatch(...)
> 11. base:::tryCatchList(expr, classes, parentenv, handlers)
> 12. base:::tryCatchOne(expr, names, parentenv, handlers[[1L]])
> 13. value[[3L]](cond)
> 14. SparkR:::captureJVMException(e, method)
> ```
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-926403952
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] karenfeng commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
karenfeng commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r729268536
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +57,31 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
Review comment:
Would 22005 (error in assignment) be a better fit?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +53,32 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:", "Schema of v2 scan: %s", "Schema of v1 relation: %s" ],
+ "sqlState" : "22023"
+ },
+ "FAILED_FIND_DATA_SOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
+ },
+ "FAILED_FORMAT_DATETIME_IN_NEW_FORMATTER" : {
+ "message" : [ "Fail to format it to '%s' in the new formatter. You can set %s to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string." ]
+ },
+ "FAILED_PARSE_DATETIME_IN_NEW_PARSER" : {
+ "message" : [ "Fail to parse '%s' in the new parser. You can set %s to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string." ]
+ },
+ "FAILED_RECOGNIZE_PATTERN_AFTER_UPGRADE" : {
Review comment:
Eg: `FAILED_RECOGNIZE_DATE_TIME_PATTERN_AFTER_UPGRADE`
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -30,16 +44,43 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:", "Schema of v2 scan: %s", "Schema of v1 relation: %s" ],
+ "sqlState" : "22023"
Review comment:
When you change this to an internal error, please remove this.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -435,131 +435,129 @@ object QueryExecutionErrors {
}
def createStreamingSourceNotSpecifySchemaError(): Throwable = {
- new IllegalArgumentException(
- s"""
- |Schema must be specified when creating a streaming source DataFrame. If some
- |files already exist in the directory, then depending on the file format you
- |may be able to create a static DataFrame on that directory with
- |'spark.read.load(directory)' and infer schema from it.
- """.stripMargin)
+ new SparkIllegalArgumentException(
+ errorClass = "MISSING_STREAMING_SOURCE_SCHEMA",
+ messageParameters = Array.empty)
}
def streamedOperatorUnsupportedByDataSourceError(
className: String, operator: String): Throwable = {
- new UnsupportedOperationException(
- s"Data source $className does not support streamed $operator")
+ new SparkUnsupportedOperationException(
+ errorClass = "UNSUPPORTED_STREAMED_OPERATOR_BY_DATA_SOURCE",
+ messageParameters = Array(className, operator))
}
def multiplePathsSpecifiedError(allPaths: Seq[String]): Throwable = {
- new IllegalArgumentException("Expected exactly one path to be specified, but " +
- s"got: ${allPaths.mkString(", ")}")
+ new SparkIllegalArgumentException(
+ errorClass = "SPECIFIED_MULTIPLE_PATHS",
+ messageParameters = Array(allPaths.mkString(", ")))
}
def failedToFindDataSourceError(provider: String, error: Throwable): Throwable = {
- new ClassNotFoundException(
- s"""
- |Failed to find data source: $provider. Please find packages at
- |http://spark.apache.org/third-party-projects.html
- """.stripMargin, error)
+ new SparkClassNotFoundException(
+ errorClass = "FAILED_FIND_DATA_SOURCE",
+ messageParameters = Array(provider), error)
}
def removedClassInSpark2Error(className: String, e: Throwable): Throwable = {
- new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
- "Please check if your library is compatible with Spark 2.0", e)
+ new SparkClassNotFoundException(
+ errorClass = "CANNOT_FIND_CLASS_IN_SPARK2",
+ messageParameters = Array(className), e)
}
def incompatibleDataSourceRegisterError(e: Throwable): Throwable = {
- new SparkClassNotFoundException("INCOMPATIBLE_DATASOURCE_REGISTER", Array(e.getMessage), e)
+ new SparkClassNotFoundException(
+ errorClass = "INCOMPATIBLE_DATASOURCE_REGISTER",
+ messageParameters = Array(e.getMessage), e)
}
def unrecognizedFileFormatError(format: String): Throwable = {
- new IllegalStateException(s"unrecognized format $format")
+ new SparkIllegalStateException(
+ errorClass = "UNRECOGNIZED_FORMAT",
+ messageParameters = Array(format))
}
def sparkUpgradeInReadingDatesError(
format: String, config: String, option: String): SparkUpgradeException = {
- new SparkUpgradeException("3.0",
- s"""
- |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
- |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
- |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
- |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
- |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
- |w.r.t. the calendar difference during reading. To read the datetime values as it is,
- |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
- """.stripMargin, null)
+ new SparkUpgradeException(
+ version = "3.0",
+ errorClass = "READING_AMBIGUOUS_DATES_AFTER_UPGRADE",
+ messageParameters = Array(format, config, option, config, option), null)
}
def sparkUpgradeInWritingDatesError(format: String, config: String): SparkUpgradeException = {
- new SparkUpgradeException("3.0",
- s"""
- |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
- |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
- |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
- |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
- |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
- |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
- |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
- |systems that use Proleptic Gregorian calendar.
- """.stripMargin, null)
+ new SparkUpgradeException(
+ version = "3.0",
+ errorClass = "WRITING_AMBIGUOUS_DATES_AFTER_UPGRADE",
+ messageParameters = Array(format, config, config), null)
}
def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
- new UnsupportedOperationException(s"buildReader is not supported for $format")
+ new SparkUnsupportedOperationException(
+ errorClass = "UNSUPPORTED_BUILD_READER_FOR_FILE_FORMAT",
+ messageParameters = Array(format))
}
def jobAbortedError(cause: Throwable): Throwable = {
- new SparkException("Job aborted.", cause)
+ new SparkException(
+ errorClass = "JOB_ABORTED",
+ messageParameters = Array.empty, cause)
}
def taskFailedWhileWritingRowsError(cause: Throwable): Throwable = {
- new SparkException("Task failed while writing rows.", cause)
+ new SparkException(
+ errorClass = "FAILED_TASK_WHILE_WRITING_ROWS",
+ messageParameters = Array.empty, cause)
}
def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
- new FileNotFoundException(
- s"""
- |${e.getMessage}\n
- |It is possible the underlying files have been updated. You can explicitly invalidate
- |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
- |recreating the Dataset/DataFrame involved.
- """.stripMargin)
+ new SparkFileNotFoundException(
+ errorClass = "CANNOT_READ_CURRENT_FILE",
+ messageParameters = Array(e.getMessage))
}
def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
- new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ new SparkIllegalStateException(
+ errorClass = "UNSUPPORTED_SAVE_MODE",
+ messageParameters = Array(saveMode + " (" + pathExists.toString + ")"))
Review comment:
Maybe we can make this into 2 different parameters? Then it would look like `%s (%s)`.
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -134,13 +181,28 @@
"message" : [ "The second argument of '%s' function needs to be an integer." ],
"sqlState" : "22023"
},
+ "SPECIFIED_MULTIPLE_PATHS" : {
+ "message" : [ "Expected exactly one path to be specified, but got: %s" ],
+ "sqlState" : "22023"
Review comment:
I think that 42000 may actually be a better fit than 22023; 22023 is a data exception. This is more of a syntax error. Can you apply this across your PR as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r730697045
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -30,16 +44,43 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:", "Schema of v2 scan: %s", "Schema of v1 relation: %s" ],
+ "sqlState" : "22023"
Review comment:
About [#comments](https://github.com/apache/spark/pull/34093#discussion_r729268536). I think `cast` don't just happen in assignments. but it is most of the scenes, So I agree with you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-945644647
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144363/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r724832293
##########
File path: core/src/main/scala/org/apache/spark/SparkException.scala
##########
@@ -72,9 +72,47 @@ private[spark] case class ExecutorDeadException(message: String)
/**
* Exception thrown when Spark returns different result after upgrading to a new version.
*/
-private[spark] class SparkUpgradeException(version: String, message: String, cause: Throwable)
+private[spark] class SparkUpgradeException(
+ version: String,
+ message: String,
+ cause: Throwable,
+ errorClass: Option[String],
+ messageParameters: Array[String])
extends RuntimeException("You may get a different result due to the upgrading of Spark" +
- s" $version: $message", cause)
+ s" $version: $message", cause) with SparkThrowable {
+
+ def this(version: String, message: String, cause: Throwable) =
+ this(
+ version = version,
+ message = message,
+ cause = cause,
+ errorClass = None,
+ messageParameters = Array.empty
+ )
+
+ def this(errorClass: String, messageParameters: Array[String], cause: Throwable) =
Review comment:
Yeah. I agree with you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r724836653
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +53,32 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:", "Schema of v2 scan: %s", "Schema of v1 relation: %s" ],
+ "sqlState" : "22023"
+ },
+ "FAILED_FIND_DATA_SOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
+ },
+ "FAILED_FORMAT_DATETIME_IN_NEW_FORMATTER" : {
Review comment:
I agree with that they can all unify as `_AFTER_UPGRADE`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r718095641
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -110,6 +153,10 @@
"message" : [ "Unknown static partition column: %s" ],
"sqlState" : "42000"
},
+ "MISSING_STREAMING_SOURCE_SCHEMA" : {
+ "message" : [ "Schema must be specified when creating a streaming source DataFrame. If some files already exist in the directory, then depending on the file format you may be able to create a static DataFrame on that directory with 'spark.read.load(directory)' and infer schema from it." ],
+ "sqlState" : "3F000"
Review comment:
Agree with you. Thank you very much.
##########
File path: R/pkg/tests/fulltests/test_sparkSQL.R
##########
@@ -3873,7 +3873,8 @@ test_that("Call DataFrameWriter.save() API in Java without path and check argume
# It makes sure that we can omit path argument in write.df API and then it calls
# DataFrameWriter.save() without path.
expect_error(write.df(df, source = "csv"),
- "Error in save : illegal argument - Expected exactly one path to be specified")
+ paste("Error in save : org.apache.spark.SparkIllegalArgumentException:",
Review comment:
Thank you for the reminder.
##########
File path: core/src/main/scala/org/apache/spark/SparkException.scala
##########
@@ -72,9 +72,11 @@ private[spark] case class ExecutorDeadException(message: String)
/**
* Exception thrown when Spark returns different result after upgrading to a new version.
*/
-private[spark] class SparkUpgradeException(version: String, message: String, cause: Throwable)
- extends RuntimeException("You may get a different result due to the upgrading of Spark" +
- s" $version: $message", cause)
+private[spark] class SparkUpgradeException(
Review comment:
I just think it's private, but it is more safe that use overload instead of modify it. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r715386015
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -134,13 +181,28 @@
"message" : [ "The second argument of '%s' function needs to be an integer." ],
"sqlState" : "22023"
},
+ "SPECIFIED_MULTIPLE_PATHS" : {
+ "message" : [ "Expected exactly one path to be specified, but got: %s" ],
+ "sqlState" : "22023"
+ },
+ "TASK_FAILED_WRITING_ROWS" : {
+ "message" : [ "Task failed while writing rows." ]
+ },
"UNABLE_TO_ACQUIRE_MEMORY" : {
"message" : [ "Unable to acquire %s bytes of memory, got %s" ]
},
+ "UNRECOGNIZED_FORMAT" : {
+ "message" : [ "unrecognized format %s" ],
+ "sqlState" : "22023"
Review comment:
invalid parameter value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r715385403
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +57,31 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
Review comment:
invalid parameter value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r724888354
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -153,13 +215,24 @@
"message" : [ "Unsupported literal type %s %s" ],
"sqlState" : "0A000"
},
+ "UNSUPPORTED_SAVE_MODE" : {
+ "message" : [ "unsupported save mode %s" ],
+ "sqlState" : "0A000"
+ },
"UNSUPPORTED_SIMPLE_STRING_WITH_NODE_ID" : {
"message" : [ "%s does not implement simpleStringWithNodeId" ]
},
+ "UNSUPPORTED_STREAMED_OPERATOR_BY_DATA_SOURCE" : {
+ "message" : [ "Data source %s does not support streamed %s" ],
+ "sqlState" : "0A000"
+ },
"UNSUPPORTED_TRANSACTION_BY_JDBC_SERVER" : {
"message" : [ "The target JDBC server does not support transaction and can only support ALTER TABLE with a single action." ],
"sqlState" : "0A000"
},
+ "WRITING_AMBIGUOUS_DATES" : {
Review comment:
I think unify error classes as `_AFTER_UPGRADE` that SparkUpgradeException is more generic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-945533364
**[Test build #144363 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144363/testReport)** for PR 34093 at commit [`e7150d8`](https://github.com/apache/spark/commit/e7150d824f55c6d97bf3b109607959c8df911c32).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-945629504
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48838/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-945533364
**[Test build #144363 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144363/testReport)** for PR 34093 at commit [`e7150d8`](https://github.com/apache/spark/commit/e7150d824f55c6d97bf3b109607959c8df911c32).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] karenfeng commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
karenfeng commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r717205325
##########
File path: R/pkg/tests/fulltests/test_sparkSQL.R
##########
@@ -3873,7 +3873,8 @@ test_that("Call DataFrameWriter.save() API in Java without path and check argume
# It makes sure that we can omit path argument in write.df API and then it calls
# DataFrameWriter.save() without path.
expect_error(write.df(df, source = "csv"),
- "Error in save : illegal argument - Expected exactly one path to be specified")
+ paste("Error in save : org.apache.spark.SparkIllegalArgumentException:",
Review comment:
Hm... It may be more correct to keep the original behavior. Can you go here https://github.com/apache/spark/blob/e024bdc30620867943b4b926f703f6a5634f9322/R/pkg/R/utils.R#L836 and add another case for `SparkIllegalArgumentException`?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
Review comment:
We can probably simplify this: CANNOT_CLEAR_SOME_DIRECTORY -> CANNOT_CLEAR_DIRECTORY
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -153,9 +215,17 @@
"message" : [ "Unsupported literal type %s %s" ],
"sqlState" : "0A000"
},
+ "UNSUPPORTED_SAVE_MODE" : {
+ "message" : [ "unsupported save mode %s" ],
+ "sqlState" : "0A000"
+ },
"UNSUPPORTED_SIMPLE_STRING_WITH_NODE_ID" : {
"message" : [ "%s does not implement simpleStringWithNodeId" ]
},
+ "UNSUPPORTED_STREAMED_OPERATOR_BY_DATASOURCE" : {
Review comment:
DATASOURCE -> DATA_SOURCE
##########
File path: core/src/main/scala/org/apache/spark/SparkException.scala
##########
@@ -72,9 +72,11 @@ private[spark] case class ExecutorDeadException(message: String)
/**
* Exception thrown when Spark returns different result after upgrading to a new version.
*/
-private[spark] class SparkUpgradeException(version: String, message: String, cause: Throwable)
- extends RuntimeException("You may get a different result due to the upgrading of Spark" +
- s" $version: $message", cause)
+private[spark] class SparkUpgradeException(
Review comment:
I'm not sure if it's safe to modify the existing constructor; can we overload it instead?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
+ },
"CANNOT_EVALUATE_EXPRESSION" : {
"message" : [ "Cannot evaluate expression: %s" ]
},
+ "CANNOT_FIND_CLASS_IN_SPARK2" : {
+ "message" : [ "%s was removed in Spark 2.0. Please check if your library is compatible with Spark 2.0" ]
+ },
"CANNOT_GENERATE_CODE_FOR_EXPRESSION" : {
"message" : [ "Cannot generate code for expression: %s" ]
},
"CANNOT_PARSE_DECIMAL" : {
"message" : [ "Cannot parse decimal" ],
"sqlState" : "42000"
},
+ "CANNOT_READ_CURRENT_FILE" : {
+ "message" : [ "%s \n It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved." ]
+ },
"CANNOT_TERMINATE_GENERATOR" : {
"message" : [ "Cannot terminate expression: %s" ]
},
+ "CANNOT_UPGRADE_IN_READING_DATES" : {
Review comment:
This is a little confusing - the issue is not that they cannot upgrade. Maybe `READING_AMBIGUOUS_DATES`?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
+ },
"CANNOT_EVALUATE_EXPRESSION" : {
"message" : [ "Cannot evaluate expression: %s" ]
},
+ "CANNOT_FIND_CLASS_IN_SPARK2" : {
+ "message" : [ "%s was removed in Spark 2.0. Please check if your library is compatible with Spark 2.0" ]
+ },
"CANNOT_GENERATE_CODE_FOR_EXPRESSION" : {
"message" : [ "Cannot generate code for expression: %s" ]
},
"CANNOT_PARSE_DECIMAL" : {
"message" : [ "Cannot parse decimal" ],
"sqlState" : "42000"
},
+ "CANNOT_READ_CURRENT_FILE" : {
+ "message" : [ "%s \n It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved." ]
+ },
"CANNOT_TERMINATE_GENERATOR" : {
"message" : [ "Cannot terminate expression: %s" ]
},
+ "CANNOT_UPGRADE_IN_READING_DATES" : {
+ "message" : [ "You may get a different result due to the upgrading of Spark %s reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from %s files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set the SQL config '%s' or the datasource option '%s' to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during reading. To read the datetime values as it is, set the SQL config '%s' or the datasource option '%s' to 'CORRECTED'." ]
+ },
+ "CANNOT_UPGRADE_IN_WRITING_DATES" : {
+ "message" : [ "You may get a different result due to the upgrading of Spark %s writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into %s files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set %s to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during writing, to get maximum interoperability. Or set %s to 'CORRECTED' to write the datetime values as it is, if you are 100%% sure that the written files will only be read by Spark 3.0+ or other systems that use Proleptic Gregorian calendar." ]
Review comment:
Missing colon here as well
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
+ },
"CANNOT_EVALUATE_EXPRESSION" : {
"message" : [ "Cannot evaluate expression: %s" ]
},
+ "CANNOT_FIND_CLASS_IN_SPARK2" : {
+ "message" : [ "%s was removed in Spark 2.0. Please check if your library is compatible with Spark 2.0" ]
+ },
"CANNOT_GENERATE_CODE_FOR_EXPRESSION" : {
"message" : [ "Cannot generate code for expression: %s" ]
},
"CANNOT_PARSE_DECIMAL" : {
"message" : [ "Cannot parse decimal" ],
"sqlState" : "42000"
},
+ "CANNOT_READ_CURRENT_FILE" : {
+ "message" : [ "%s \n It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved." ]
+ },
"CANNOT_TERMINATE_GENERATOR" : {
"message" : [ "Cannot terminate expression: %s" ]
},
+ "CANNOT_UPGRADE_IN_READING_DATES" : {
+ "message" : [ "You may get a different result due to the upgrading of Spark %s reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from %s files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set the SQL config '%s' or the datasource option '%s' to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during reading. To read the datetime values as it is, set the SQL config '%s' or the datasource option '%s' to 'CORRECTED'." ]
+ },
+ "CANNOT_UPGRADE_IN_WRITING_DATES" : {
Review comment:
WRITING_AMBIGUOUS_DATES may be a better descriptor
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
+ },
"CANNOT_EVALUATE_EXPRESSION" : {
"message" : [ "Cannot evaluate expression: %s" ]
},
+ "CANNOT_FIND_CLASS_IN_SPARK2" : {
+ "message" : [ "%s was removed in Spark 2.0. Please check if your library is compatible with Spark 2.0" ]
+ },
"CANNOT_GENERATE_CODE_FOR_EXPRESSION" : {
"message" : [ "Cannot generate code for expression: %s" ]
},
"CANNOT_PARSE_DECIMAL" : {
"message" : [ "Cannot parse decimal" ],
"sqlState" : "42000"
},
+ "CANNOT_READ_CURRENT_FILE" : {
+ "message" : [ "%s \n It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved." ]
Review comment:
Rather than use a literal `\n` in the message, you can split up the message into separate elements in the array and they'll be joined with newlines later:
```
"message" : [ "%s", "It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved." ]
```
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
+ },
"CANNOT_EVALUATE_EXPRESSION" : {
"message" : [ "Cannot evaluate expression: %s" ]
},
+ "CANNOT_FIND_CLASS_IN_SPARK2" : {
+ "message" : [ "%s was removed in Spark 2.0. Please check if your library is compatible with Spark 2.0" ]
+ },
"CANNOT_GENERATE_CODE_FOR_EXPRESSION" : {
"message" : [ "Cannot generate code for expression: %s" ]
},
"CANNOT_PARSE_DECIMAL" : {
"message" : [ "Cannot parse decimal" ],
"sqlState" : "42000"
},
+ "CANNOT_READ_CURRENT_FILE" : {
+ "message" : [ "%s \n It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved." ]
+ },
"CANNOT_TERMINATE_GENERATOR" : {
"message" : [ "Cannot terminate expression: %s" ]
},
+ "CANNOT_UPGRADE_IN_READING_DATES" : {
+ "message" : [ "You may get a different result due to the upgrading of Spark %s reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from %s files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set the SQL config '%s' or the datasource option '%s' to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during reading. To read the datetime values as it is, set the SQL config '%s' or the datasource option '%s' to 'CORRECTED'." ]
Review comment:
There was a colon here before: `Spark %s: reading dates`
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +57,31 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:\n Schema of v2 scan: %s\nSchema of v1 relation: %s" ]
Review comment:
Split strings into array elements
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
+ },
"CANNOT_EVALUATE_EXPRESSION" : {
"message" : [ "Cannot evaluate expression: %s" ]
},
+ "CANNOT_FIND_CLASS_IN_SPARK2" : {
+ "message" : [ "%s was removed in Spark 2.0. Please check if your library is compatible with Spark 2.0" ]
Review comment:
I would consider this to be an unsupported operation with SQLSTATE 0A000
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
Review comment:
I would consider this a syntax error; can you add SQLSTATE 42000?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +57,31 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:\n Schema of v2 scan: %s\nSchema of v1 relation: %s" ]
+ },
+ "FAILED_FIND_DATASOURCE" : {
Review comment:
DATASOURCE -> DATA_SOURCE
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -110,6 +153,10 @@
"message" : [ "Unknown static partition column: %s" ],
"sqlState" : "42000"
},
+ "MISSING_STREAMING_SOURCE_SCHEMA" : {
+ "message" : [ "Schema must be specified when creating a streaming source DataFrame. If some files already exist in the directory, then depending on the file format you may be able to create a static DataFrame on that directory with 'spark.read.load(directory)' and infer schema from it." ],
+ "sqlState" : "3F000"
Review comment:
Is this the same schema as intended in the SQLSTATE? It seems like that "schema" is referring to schema objects. Maybe this should be 22023?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +57,31 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:\n Schema of v2 scan: %s\nSchema of v1 relation: %s" ]
+ },
+ "FAILED_FIND_DATASOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
Review comment:
Would this have a SQLSTATE like 22023?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-943085144
**[Test build #144234 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144234/testReport)** for PR 34093 at commit [`c8c16ad`](https://github.com/apache/spark/commit/c8c16ad50e01c6c5d247a0b6a1c15c07f9f92010).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-942931630
**[Test build #144234 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144234/testReport)** for PR 34093 at commit [`c8c16ad`](https://github.com/apache/spark/commit/c8c16ad50e01c6c5d247a0b6a1c15c07f9f92010).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-927223084
Thanks for working on this @Peng-Lei
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-927223051
Seems related test failure:
```
`write.df(df, source = "csv")` threw an error with unexpected message.
Expected match: "Error in save : org.apache.spark.SparkIllegalArgumentException: Expected exactly one path to be specified"
Actual message: "Error in save : org.apache.spark.SparkIllegalArgumentException: Expected exactly one path to be specified, but got: \n\tat org.apache.spark.sql.errors.QueryExecutionErrors$.multiplePathsSpecifiedError(QueryExecutionErrors.scala:450)\n\tat org.apache.spark.sql.execution.datasources.DataSource.planForWritingFileFormat(DataSource.scala:464)\n\tat org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:558)\n\tat org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:382)\n\tat org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:355)\n\tat org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:247)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n
\tat org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:164)\n\tat org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:105)\n\tat org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:39)\n\tat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\
n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerC
ontext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecuto
r.java:986)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.lang.Thread.run(Thread.java:748)\n\n"
Backtrace:
1. testthat::expect_error(...) test_sparkSQL.R:3875:2
7. SparkR::write.df(df, source = "csv")
8. SparkR:::.local(df, path, ...)
9. SparkR:::handledCallJMethod(write, "save")
10. base::tryCatch(...)
11. base:::tryCatchList(expr, classes, parentenv, handlers)
12. base:::tryCatchOne(expr, names, parentenv, handlers[[1L]])
13. value[[3L]](cond)
14. SparkR:::captureJVMException(e, method)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] karenfeng commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
karenfeng commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r720417700
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +53,32 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:", "Schema of v2 scan: %s", "Schema of v1 relation: %s" ],
+ "sqlState" : "22023"
+ },
+ "FAILED_FIND_DATA_SOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
+ },
+ "FAILED_FORMAT_DATETIME_IN_NEW_FORMATTER" : {
Review comment:
Alternately, rather than provide the explicit Spark version in the class name, they can all unify as `_AFTER_UPGRADE`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] karenfeng commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
karenfeng commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r726664631
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +53,32 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:", "Schema of v2 scan: %s", "Schema of v1 relation: %s" ],
+ "sqlState" : "22023"
+ },
+ "FAILED_FIND_DATA_SOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
+ },
+ "FAILED_FORMAT_DATETIME_IN_NEW_FORMATTER" : {
+ "message" : [ "Fail to format it to '%s' in the new formatter. You can set %s to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string." ]
+ },
+ "FAILED_PARSE_DATETIME_IN_NEW_PARSER" : {
+ "message" : [ "Fail to parse '%s' in the new parser. You can set %s to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string." ]
+ },
+ "FAILED_RECOGNIZE_PATTERN_AFTER_UPGRADE" : {
Review comment:
Let's make it clear that this is about a date time formatter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-942917521
@karenfeng Thank you look again. I address the comments except [#comment](https://github.com/apache/spark/pull/34093#discussion_r726629302). And add sqlState for [FAILED_FIND_DATA_SOURCE](https://github.com/apache/spark/pull/34093#discussion_r726628407)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-943086979
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144234/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-930348735
@karenfeng Done. Cloud you take a look? Thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] karenfeng commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
karenfeng commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r717205325
##########
File path: R/pkg/tests/fulltests/test_sparkSQL.R
##########
@@ -3873,7 +3873,8 @@ test_that("Call DataFrameWriter.save() API in Java without path and check argume
# It makes sure that we can omit path argument in write.df API and then it calls
# DataFrameWriter.save() without path.
expect_error(write.df(df, source = "csv"),
- "Error in save : illegal argument - Expected exactly one path to be specified")
+ paste("Error in save : org.apache.spark.SparkIllegalArgumentException:",
Review comment:
Hm... It may be more correct to keep the original behavior. Can you go here https://github.com/apache/spark/blob/e024bdc30620867943b4b926f703f6a5634f9322/R/pkg/R/utils.R#L836 and add another case for `SparkIllegalArgumentException`?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
Review comment:
We can probably simplify this: CANNOT_CLEAR_SOME_DIRECTORY -> CANNOT_CLEAR_DIRECTORY
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -153,9 +215,17 @@
"message" : [ "Unsupported literal type %s %s" ],
"sqlState" : "0A000"
},
+ "UNSUPPORTED_SAVE_MODE" : {
+ "message" : [ "unsupported save mode %s" ],
+ "sqlState" : "0A000"
+ },
"UNSUPPORTED_SIMPLE_STRING_WITH_NODE_ID" : {
"message" : [ "%s does not implement simpleStringWithNodeId" ]
},
+ "UNSUPPORTED_STREAMED_OPERATOR_BY_DATASOURCE" : {
Review comment:
DATASOURCE -> DATA_SOURCE
##########
File path: core/src/main/scala/org/apache/spark/SparkException.scala
##########
@@ -72,9 +72,11 @@ private[spark] case class ExecutorDeadException(message: String)
/**
* Exception thrown when Spark returns different result after upgrading to a new version.
*/
-private[spark] class SparkUpgradeException(version: String, message: String, cause: Throwable)
- extends RuntimeException("You may get a different result due to the upgrading of Spark" +
- s" $version: $message", cause)
+private[spark] class SparkUpgradeException(
Review comment:
I'm not sure if it's safe to modify the existing constructor; can we overload it instead?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
+ },
"CANNOT_EVALUATE_EXPRESSION" : {
"message" : [ "Cannot evaluate expression: %s" ]
},
+ "CANNOT_FIND_CLASS_IN_SPARK2" : {
+ "message" : [ "%s was removed in Spark 2.0. Please check if your library is compatible with Spark 2.0" ]
+ },
"CANNOT_GENERATE_CODE_FOR_EXPRESSION" : {
"message" : [ "Cannot generate code for expression: %s" ]
},
"CANNOT_PARSE_DECIMAL" : {
"message" : [ "Cannot parse decimal" ],
"sqlState" : "42000"
},
+ "CANNOT_READ_CURRENT_FILE" : {
+ "message" : [ "%s \n It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved." ]
+ },
"CANNOT_TERMINATE_GENERATOR" : {
"message" : [ "Cannot terminate expression: %s" ]
},
+ "CANNOT_UPGRADE_IN_READING_DATES" : {
Review comment:
This is a little confusing - the issue is not that they cannot upgrade. Maybe `READING_AMBIGUOUS_DATES`?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
+ },
"CANNOT_EVALUATE_EXPRESSION" : {
"message" : [ "Cannot evaluate expression: %s" ]
},
+ "CANNOT_FIND_CLASS_IN_SPARK2" : {
+ "message" : [ "%s was removed in Spark 2.0. Please check if your library is compatible with Spark 2.0" ]
+ },
"CANNOT_GENERATE_CODE_FOR_EXPRESSION" : {
"message" : [ "Cannot generate code for expression: %s" ]
},
"CANNOT_PARSE_DECIMAL" : {
"message" : [ "Cannot parse decimal" ],
"sqlState" : "42000"
},
+ "CANNOT_READ_CURRENT_FILE" : {
+ "message" : [ "%s \n It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved." ]
+ },
"CANNOT_TERMINATE_GENERATOR" : {
"message" : [ "Cannot terminate expression: %s" ]
},
+ "CANNOT_UPGRADE_IN_READING_DATES" : {
+ "message" : [ "You may get a different result due to the upgrading of Spark %s reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from %s files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set the SQL config '%s' or the datasource option '%s' to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during reading. To read the datetime values as it is, set the SQL config '%s' or the datasource option '%s' to 'CORRECTED'." ]
+ },
+ "CANNOT_UPGRADE_IN_WRITING_DATES" : {
+ "message" : [ "You may get a different result due to the upgrading of Spark %s writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into %s files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set %s to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during writing, to get maximum interoperability. Or set %s to 'CORRECTED' to write the datetime values as it is, if you are 100%% sure that the written files will only be read by Spark 3.0+ or other systems that use Proleptic Gregorian calendar." ]
Review comment:
Missing colon here as well
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
+ },
"CANNOT_EVALUATE_EXPRESSION" : {
"message" : [ "Cannot evaluate expression: %s" ]
},
+ "CANNOT_FIND_CLASS_IN_SPARK2" : {
+ "message" : [ "%s was removed in Spark 2.0. Please check if your library is compatible with Spark 2.0" ]
+ },
"CANNOT_GENERATE_CODE_FOR_EXPRESSION" : {
"message" : [ "Cannot generate code for expression: %s" ]
},
"CANNOT_PARSE_DECIMAL" : {
"message" : [ "Cannot parse decimal" ],
"sqlState" : "42000"
},
+ "CANNOT_READ_CURRENT_FILE" : {
+ "message" : [ "%s \n It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved." ]
+ },
"CANNOT_TERMINATE_GENERATOR" : {
"message" : [ "Cannot terminate expression: %s" ]
},
+ "CANNOT_UPGRADE_IN_READING_DATES" : {
+ "message" : [ "You may get a different result due to the upgrading of Spark %s reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from %s files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set the SQL config '%s' or the datasource option '%s' to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during reading. To read the datetime values as it is, set the SQL config '%s' or the datasource option '%s' to 'CORRECTED'." ]
+ },
+ "CANNOT_UPGRADE_IN_WRITING_DATES" : {
Review comment:
WRITING_AMBIGUOUS_DATES may be a better descriptor
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
+ },
"CANNOT_EVALUATE_EXPRESSION" : {
"message" : [ "Cannot evaluate expression: %s" ]
},
+ "CANNOT_FIND_CLASS_IN_SPARK2" : {
+ "message" : [ "%s was removed in Spark 2.0. Please check if your library is compatible with Spark 2.0" ]
+ },
"CANNOT_GENERATE_CODE_FOR_EXPRESSION" : {
"message" : [ "Cannot generate code for expression: %s" ]
},
"CANNOT_PARSE_DECIMAL" : {
"message" : [ "Cannot parse decimal" ],
"sqlState" : "42000"
},
+ "CANNOT_READ_CURRENT_FILE" : {
+ "message" : [ "%s \n It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved." ]
Review comment:
Rather than use a literal `\n` in the message, you can split up the message into separate elements in the array and they'll be joined with newlines later:
```
"message" : [ "%s", "It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved." ]
```
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
+ },
"CANNOT_EVALUATE_EXPRESSION" : {
"message" : [ "Cannot evaluate expression: %s" ]
},
+ "CANNOT_FIND_CLASS_IN_SPARK2" : {
+ "message" : [ "%s was removed in Spark 2.0. Please check if your library is compatible with Spark 2.0" ]
+ },
"CANNOT_GENERATE_CODE_FOR_EXPRESSION" : {
"message" : [ "Cannot generate code for expression: %s" ]
},
"CANNOT_PARSE_DECIMAL" : {
"message" : [ "Cannot parse decimal" ],
"sqlState" : "42000"
},
+ "CANNOT_READ_CURRENT_FILE" : {
+ "message" : [ "%s \n It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved." ]
+ },
"CANNOT_TERMINATE_GENERATOR" : {
"message" : [ "Cannot terminate expression: %s" ]
},
+ "CANNOT_UPGRADE_IN_READING_DATES" : {
+ "message" : [ "You may get a different result due to the upgrading of Spark %s reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from %s files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set the SQL config '%s' or the datasource option '%s' to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during reading. To read the datetime values as it is, set the SQL config '%s' or the datasource option '%s' to 'CORRECTED'." ]
Review comment:
There was a colon here before: `Spark %s: reading dates`
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +57,31 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:\n Schema of v2 scan: %s\nSchema of v1 relation: %s" ]
Review comment:
Split strings into array elements
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
+ },
"CANNOT_EVALUATE_EXPRESSION" : {
"message" : [ "Cannot evaluate expression: %s" ]
},
+ "CANNOT_FIND_CLASS_IN_SPARK2" : {
+ "message" : [ "%s was removed in Spark 2.0. Please check if your library is compatible with Spark 2.0" ]
Review comment:
I would consider this to be an unsupported operation with SQLSTATE 0A000
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -11,19 +11,37 @@
"message" : [ "%s cannot be represented as Decimal(%s, %s)." ],
"sqlState" : "22005"
},
+ "CANNOT_CLEAR_SOME_DIRECTORY" : {
+ "message" : [ "Unable to clear %s directory %s prior to writing to it" ]
+ },
+ "CANNOT_DROP_NONEMPTY_NAMESPACE" : {
+ "message" : [ "Cannot drop a non-empty namespace: %s. Use CASCADE option to drop a non-empty namespace." ]
Review comment:
I would consider this a syntax error; can you add SQLSTATE 42000?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +57,31 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:\n Schema of v2 scan: %s\nSchema of v1 relation: %s" ]
+ },
+ "FAILED_FIND_DATASOURCE" : {
Review comment:
DATASOURCE -> DATA_SOURCE
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -110,6 +153,10 @@
"message" : [ "Unknown static partition column: %s" ],
"sqlState" : "42000"
},
+ "MISSING_STREAMING_SOURCE_SCHEMA" : {
+ "message" : [ "Schema must be specified when creating a streaming source DataFrame. If some files already exist in the directory, then depending on the file format you may be able to create a static DataFrame on that directory with 'spark.read.load(directory)' and infer schema from it." ],
+ "sqlState" : "3F000"
Review comment:
Is this the same schema as intended in the SQLSTATE? It seems like that "schema" is referring to schema objects. Maybe this should be 22023?
##########
File path: core/src/main/resources/error/error-classes.json
##########
@@ -39,9 +57,31 @@
"message" : [ "Found duplicate keys '%s'" ],
"sqlState" : "23000"
},
+ "END_OF_STREAM" : {
+ "message" : [ "End of stream" ]
+ },
+ "FAILED_CAST_VALUE_TO_DATATYPE_FOR_PARTITION_COLUMN" : {
+ "message" : [ "Failed to cast value `%s` to `%s` for partition column `%s`" ],
+ "sqlState" : "22023"
+ },
"FAILED_EXECUTE_UDF" : {
"message" : [ "Failed to execute user defined function (%s: (%s) => %s)" ]
},
+ "FAILED_FALLBACK_V1_BECAUSE_OF_INCONSISTENT_SCHEMA" : {
+ "message" : [ "The fallback v1 relation reports inconsistent schema:\n Schema of v2 scan: %s\nSchema of v1 relation: %s" ]
+ },
+ "FAILED_FIND_DATASOURCE" : {
+ "message" : [ "Failed to find data source: %s. Please find packages at http://spark.apache.org/third-party-projects.html" ]
Review comment:
Would this have a SQLSTATE like 22023?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34093:
URL: https://github.com/apache/spark/pull/34093#issuecomment-945630946
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48838/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34093:
URL: https://github.com/apache/spark/pull/34093#discussion_r730639076
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -435,131 +435,129 @@ object QueryExecutionErrors {
}
def createStreamingSourceNotSpecifySchemaError(): Throwable = {
- new IllegalArgumentException(
- s"""
- |Schema must be specified when creating a streaming source DataFrame. If some
- |files already exist in the directory, then depending on the file format you
- |may be able to create a static DataFrame on that directory with
- |'spark.read.load(directory)' and infer schema from it.
- """.stripMargin)
+ new SparkIllegalArgumentException(
+ errorClass = "MISSING_STREAMING_SOURCE_SCHEMA",
+ messageParameters = Array.empty)
}
def streamedOperatorUnsupportedByDataSourceError(
className: String, operator: String): Throwable = {
- new UnsupportedOperationException(
- s"Data source $className does not support streamed $operator")
+ new SparkUnsupportedOperationException(
+ errorClass = "UNSUPPORTED_STREAMED_OPERATOR_BY_DATA_SOURCE",
+ messageParameters = Array(className, operator))
}
def multiplePathsSpecifiedError(allPaths: Seq[String]): Throwable = {
- new IllegalArgumentException("Expected exactly one path to be specified, but " +
- s"got: ${allPaths.mkString(", ")}")
+ new SparkIllegalArgumentException(
+ errorClass = "SPECIFIED_MULTIPLE_PATHS",
+ messageParameters = Array(allPaths.mkString(", ")))
}
def failedToFindDataSourceError(provider: String, error: Throwable): Throwable = {
- new ClassNotFoundException(
- s"""
- |Failed to find data source: $provider. Please find packages at
- |http://spark.apache.org/third-party-projects.html
- """.stripMargin, error)
+ new SparkClassNotFoundException(
+ errorClass = "FAILED_FIND_DATA_SOURCE",
+ messageParameters = Array(provider), error)
}
def removedClassInSpark2Error(className: String, e: Throwable): Throwable = {
- new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
- "Please check if your library is compatible with Spark 2.0", e)
+ new SparkClassNotFoundException(
+ errorClass = "CANNOT_FIND_CLASS_IN_SPARK2",
+ messageParameters = Array(className), e)
}
def incompatibleDataSourceRegisterError(e: Throwable): Throwable = {
- new SparkClassNotFoundException("INCOMPATIBLE_DATASOURCE_REGISTER", Array(e.getMessage), e)
+ new SparkClassNotFoundException(
+ errorClass = "INCOMPATIBLE_DATASOURCE_REGISTER",
+ messageParameters = Array(e.getMessage), e)
}
def unrecognizedFileFormatError(format: String): Throwable = {
- new IllegalStateException(s"unrecognized format $format")
+ new SparkIllegalStateException(
+ errorClass = "UNRECOGNIZED_FORMAT",
+ messageParameters = Array(format))
}
def sparkUpgradeInReadingDatesError(
format: String, config: String, option: String): SparkUpgradeException = {
- new SparkUpgradeException("3.0",
- s"""
- |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
- |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
- |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
- |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
- |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
- |w.r.t. the calendar difference during reading. To read the datetime values as it is,
- |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
- """.stripMargin, null)
+ new SparkUpgradeException(
+ version = "3.0",
+ errorClass = "READING_AMBIGUOUS_DATES_AFTER_UPGRADE",
+ messageParameters = Array(format, config, option, config, option), null)
}
def sparkUpgradeInWritingDatesError(format: String, config: String): SparkUpgradeException = {
- new SparkUpgradeException("3.0",
- s"""
- |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
- |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
- |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
- |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
- |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
- |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
- |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
- |systems that use Proleptic Gregorian calendar.
- """.stripMargin, null)
+ new SparkUpgradeException(
+ version = "3.0",
+ errorClass = "WRITING_AMBIGUOUS_DATES_AFTER_UPGRADE",
+ messageParameters = Array(format, config, config), null)
}
def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
- new UnsupportedOperationException(s"buildReader is not supported for $format")
+ new SparkUnsupportedOperationException(
+ errorClass = "UNSUPPORTED_BUILD_READER_FOR_FILE_FORMAT",
+ messageParameters = Array(format))
}
def jobAbortedError(cause: Throwable): Throwable = {
- new SparkException("Job aborted.", cause)
+ new SparkException(
+ errorClass = "JOB_ABORTED",
+ messageParameters = Array.empty, cause)
}
def taskFailedWhileWritingRowsError(cause: Throwable): Throwable = {
- new SparkException("Task failed while writing rows.", cause)
+ new SparkException(
+ errorClass = "FAILED_TASK_WHILE_WRITING_ROWS",
+ messageParameters = Array.empty, cause)
}
def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
- new FileNotFoundException(
- s"""
- |${e.getMessage}\n
- |It is possible the underlying files have been updated. You can explicitly invalidate
- |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
- |recreating the Dataset/DataFrame involved.
- """.stripMargin)
+ new SparkFileNotFoundException(
+ errorClass = "CANNOT_READ_CURRENT_FILE",
+ messageParameters = Array(e.getMessage))
}
def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
- new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ new SparkIllegalStateException(
+ errorClass = "UNSUPPORTED_SAVE_MODE",
+ messageParameters = Array(saveMode + " (" + pathExists.toString + ")"))
Review comment:
ok, 2 ddifferent parameters looks more better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org