You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/05 09:17:04 UTC
[GitHub] [spark] beliefer opened a new pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
beliefer opened a new pull request #31757:
URL: https://github.com/apache/spark/pull/31757
### What changes were proposed in this pull request?
This PR group exception messages in `/core/src/main/scala/org/apache/spark/sql/execution/datasources`.
### Why are the changes needed?
It will largely help with standardization of error messages and its maintenance.
### Does this PR introduce _any_ user-facing change?
No. Error messages remain unchanged.
### How was this patch tested?
No new tests - pass all original tests to make sure it doesn't break any existing behavior.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-791511032
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135810/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-796511686
**[Test build #135956 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135956/testReport)** for PR 31757 at commit [`b575885`](https://github.com/apache/spark/commit/b575885942b813eabb188a9b3646ffb652e9bc87).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-799055768
cc @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a change in pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #31757:
URL: https://github.com/apache/spark/pull/31757#discussion_r590787466
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
Review comment:
dataSourceOutputModeUnsupportedError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
Review comment:
insertMismatchedColumnNumberError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
+ new AnalysisException("Can only write data to relations with a single path.")
+ }
+
+ def failedRebuildExpressionCausedByMissingKeyError(filter: Filter): Throwable = {
+ new AnalysisException(
+ s"Fail to rebuild expression: missing key $filter in `translatedFilterToExpr`")
+ }
+
+ def dataTypeUnsupportedByDataSourceError(format: String, field: StructField): Throwable = {
+ new AnalysisException(
+ s"$format data source does not support ${field.dataType.catalogString} data type.")
+ }
+
+ def failToResolveDataSourceForTableError(table: CatalogTable, key: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Fail to resolve data source for the table ${table.identifier} since the table
+ |serde property has the duplicated key $key with extra options specified for this
+ |scan operation. To fix this, you can rollback to the legacy behavior of ignoring
+ |the extra options by setting the config
+ |${SQLConf.LEGACY_EXTRA_OPTIONS_BEHAVIOR.key} to `false`, or address the
+ |conflicts of the same config.
+ """.stripMargin)
+ }
+
+ def insertIntoExistingOutputPathError(outputPath: Path): Throwable = {
+ new AnalysisException(s"path $outputPath already exists.")
+ }
+
+ def cannotUseDataTypeForPartitionColumnError(field: StructField): Throwable = {
+ new AnalysisException(s"Cannot use ${field.dataType} for partition column")
+ }
+
+ def cannotUseAllColumnsForPartitionColumnsError(): Throwable = {
+ new AnalysisException(s"Cannot use all columns for partition columns")
+ }
+
+ def partitionColumnNotFoundInSchemaError(col: String, schemaCatalog: String): Throwable = {
+ new AnalysisException(s"Partition column `$col` not found in schema $schemaCatalog")
+ }
+
+ def columnNotFoundInSchemaError(
+ col: StructField, tableSchema: Option[StructType]): Throwable = {
+ new AnalysisException(s"""Column "${col.name}" not found in schema $tableSchema""")
+ }
+
+ def unsupportedDataSourceTypeForDirectQueryOnFilesError(className: String): Throwable = {
+ new AnalysisException(s"Unsupported data source type for direct query on files: $className")
+ }
+
+ def saveDataIntoViewNotAllowedError(): Throwable = {
+ new AnalysisException("Saving data into a view is not allowed.")
+ }
+
+ def tableFormatMisMatchSpecifyOneError(
+ tableName: String, existingProvider: Class[_], specifiedProvider: Class[_]): Throwable = {
+ new AnalysisException(
+ s"""
+ |The format of the existing table $tableName is `${existingProvider.getSimpleName}`.
+ |It doesn't match the specified format `${specifiedProvider.getSimpleName}`.
+ """.stripMargin)
+ }
+
+ def tableLocationMisMatchSpecifyOneError(
+ identifier: TableIdentifier,
+ existingTable: CatalogTable,
+ tableDesc: CatalogTable): Throwable = {
+ new AnalysisException(
+ s"""
+ |The location of the existing table ${identifier.quotedString} is
+ |`${existingTable.location}`. It doesn't match the specified location
+ |`${tableDesc.location}`.
+ """.stripMargin)
+ }
+
+ def tableColumnNumberMisMatchSpecifyOneError(
+ tableName: String,
+ existingTable: CatalogTable,
+ query: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"""
+ |The column number of the existing table $tableName
+ |(${existingTable.schema.catalogString}) doesn't match the data schema
+ |(${query.schema.catalogString})
+ """.stripMargin)
+ }
+
+ def cannotResolveColumnGivenInputColumnsError(col: String, inputColumns: String): Throwable = {
+ new AnalysisException(s"cannot resolve '$col' given input columns: [$inputColumns]")
+ }
+
+ def tablePartitionMisMatchSpecifyOneError(
+ tableName: String,
+ specifiedPartCols: Seq[String],
+ existingPartCols: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Specified partitioning does not match that of the existing table $tableName.
+ |Specified partition columns: [${specifiedPartCols.mkString(", ")}]
+ |Existing partition columns: [$existingPartCols]
+ """.stripMargin)
+ }
+
+ def tableBucketMisMatchSpecifyOneError(
Review comment:
mismatchedTableBucketingError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
Review comment:
failedToFindAvroDataSourceError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
+ new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ }
+
+ def unableClearPartitionDirectoryError(path: Path): Throwable = {
+ new IOException(s"Unable to clear partition directory $path prior to writing to it")
+ }
+
+ def failedCastValueToDataTypeForPartitionColumnError(
Review comment:
failedToCastValueToDataTypeForPartitionColumnError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
+ new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ }
+
+ def unableClearPartitionDirectoryError(path: Path): Throwable = {
+ new IOException(s"Unable to clear partition directory $path prior to writing to it")
+ }
+
+ def failedCastValueToDataTypeForPartitionColumnError(
+ value: String, dataType: DataType, columnName: String): Throwable = {
+ new RuntimeException(s"Failed to cast value `$value` to " +
+ s"`$dataType` for partition column `$columnName`")
+ }
+
+ def typeUnsupportedError(dataType: DataType): Throwable = {
+ new IllegalArgumentException(s"Unexpected type $dataType")
+ }
+
+ def endOfStreamError(): Throwable = {
+ new java.util.NoSuchElementException("End of stream")
Review comment:
nit: no need for `java.util.`
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
Review comment:
failedToFindAvroDataSourceError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
+ new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ }
+
+ def unableClearPartitionDirectoryError(path: Path): Throwable = {
+ new IOException(s"Unable to clear partition directory $path prior to writing to it")
+ }
+
+ def failedCastValueToDataTypeForPartitionColumnError(
+ value: String, dataType: DataType, columnName: String): Throwable = {
+ new RuntimeException(s"Failed to cast value `$value` to " +
+ s"`$dataType` for partition column `$columnName`")
+ }
+
+ def typeUnsupportedError(dataType: DataType): Throwable = {
+ new IllegalArgumentException(s"Unexpected type $dataType")
+ }
+
+ def endOfStreamError(): Throwable = {
+ new java.util.NoSuchElementException("End of stream")
+ }
+
+ def writeUnsupportedForBinaryFileDataSourceError(): Throwable = {
+ new UnsupportedOperationException("Write is not supported for binary file data source")
+ }
+
+ def fileLengthExceedsMaxLengthError(status: FileStatus, maxLength: Int): Throwable = {
+ new SparkException(
+ s"The length of ${status.getPath} is ${status.getLen}, " +
+ s"which exceeds the max length allowed: ${maxLength}.")
+ }
+
+ def unsupportedFieldNameError(fieldName: String): Throwable = {
+ new RuntimeException(s"Unsupported field name: ${fieldName}")
+ }
+
+ def cannotSpecifyBothTableNameAndQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Both '$jdbcTableName' and '$jdbcQueryString' can not be specified at the same time.")
+ }
+
+ def notSpecifyTableNameOrQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Option '$jdbcTableName' or '$jdbcQueryString' is required."
+ )
+ }
+
+ def optionIsEmptyError(optionName: String): Throwable = {
+ new IllegalArgumentException(s"Option `$optionName` can not be empty.")
+ }
+
+ def invalidJdbcTxnIsolationLevelError(jdbcTxnIsolationLevel: String, value: String): Throwable = {
+ new IllegalArgumentException(
+ s"Invalid value `$value` for parameter `$jdbcTxnIsolationLevel`. This can be " +
+ "`NONE`, `READ_UNCOMMITTED`, `READ_COMMITTED`, `REPEATABLE_READ` or `SERIALIZABLE`.")
+ }
+
+ def cannotGetJdbcTypeError(dt: DataType): Throwable = {
+ new IllegalArgumentException(s"Can't get JDBC type for ${dt.catalogString}")
+ }
+
+ def unrecognizedSqlTypeError(sqlType: Int): Throwable = {
+ new SQLException(s"Unrecognized SQL type $sqlType")
+ }
+
+ def unrecognizedTypeError(content: String): Throwable = {
+ new SQLException(s"Unsupported type $content")
+ }
+
+ def unsupportedArrayElementTypeBasedOnBinaryError(dt: DataType): Throwable = {
+ new IllegalArgumentException(s"Unsupported array element " +
+ s"type ${dt.catalogString} based on binary")
+ }
+
+ def nestedArraysUnsupportedError(): Throwable = {
+ new IllegalArgumentException("Nested arrays unsupported")
+ }
+
+ def cannotTranslateNonNullValueForFieldError(pos: Int): Throwable = {
+ new IllegalArgumentException(s"Can't translate non-null value for field $pos")
+ }
+
+ def invalidValueForJdbcNumPartitionsError(n: Int, jdbcNumPartitions: String): Throwable = {
+ new IllegalArgumentException(
+ s"Invalid value `$n` for parameter `$jdbcNumPartitions` in table writing " +
+ "via JDBC. The minimum value is 1.")
+ }
+
+ def transactionUnsupportedByJdbcServerError(): Throwable = {
+ new SQLFeatureNotSupportedException("The target JDBC server does not support " +
+ "transaction and can only support ALTER TABLE with a single action.")
+ }
+
+ def dataTypeUnsupportedYetError(dataType: DataType): Throwable = {
+ new UnsupportedOperationException(s"$dataType is not supported yet.")
+ }
+
+ def unsupportedOperationForDataTypeError(dataType: DataType): Throwable = {
+ new UnsupportedOperationException(s"DataType: ${dataType.catalogString}")
+ }
+
+ def inputFilterNotFullyConvertibleError(own: String): Throwable = {
+ new SparkException(s"The input filter of $own should be fully convertible.")
+ }
+
+ def cannotReadFooterForFileError(file: Path, e: IOException): Throwable = {
+ new SparkException(s"Could not read footer for file: $file", e)
+ }
+
+ def cannotReadFooterForFileError(file: FileStatus, e: RuntimeException): Throwable = {
+ new Exception(s"Could not read footer for file: $file", e)
+ }
+
+ def foundDuplicateFieldInCaseInsensitiveModeError(
+ requiredFieldName: String, matchedOrcFields: String): Throwable = {
+ new RuntimeException(
+ s"""
+ |Found duplicate field(s) "$requiredFieldName": $matchedOrcFields
+ |in case-insensitive mode
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedMergeIncompatibleSchemasError(
Review comment:
failedToMergeIncompatibleSchemasError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
Review comment:
multiplePartitionColumnValuesSpecifiedError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
Review comment:
schemaNotSpecifiedForSchemaRelationProviderError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
Review comment:
invalidOrderingForConstantValuePartitionColumnError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
Review comment:
insertMismatchedPartitionNumberError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
+ new AnalysisException("Can only write data to relations with a single path.")
+ }
+
+ def failedRebuildExpressionCausedByMissingKeyError(filter: Filter): Throwable = {
Review comment:
failedToRebuildExpressionError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
Review comment:
cannotResolveAttributeError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
Review comment:
cannotWriteDataToRelationsWithMultiplePathsError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
+ new AnalysisException("Can only write data to relations with a single path.")
+ }
+
+ def failedRebuildExpressionCausedByMissingKeyError(filter: Filter): Throwable = {
+ new AnalysisException(
+ s"Fail to rebuild expression: missing key $filter in `translatedFilterToExpr`")
+ }
+
+ def dataTypeUnsupportedByDataSourceError(format: String, field: StructField): Throwable = {
+ new AnalysisException(
+ s"$format data source does not support ${field.dataType.catalogString} data type.")
+ }
+
+ def failToResolveDataSourceForTableError(table: CatalogTable, key: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Fail to resolve data source for the table ${table.identifier} since the table
+ |serde property has the duplicated key $key with extra options specified for this
+ |scan operation. To fix this, you can rollback to the legacy behavior of ignoring
+ |the extra options by setting the config
+ |${SQLConf.LEGACY_EXTRA_OPTIONS_BEHAVIOR.key} to `false`, or address the
+ |conflicts of the same config.
+ """.stripMargin)
+ }
+
+ def insertIntoExistingOutputPathError(outputPath: Path): Throwable = {
Review comment:
outputPathAlreadyExistsError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
+ new AnalysisException("Can only write data to relations with a single path.")
+ }
+
+ def failedRebuildExpressionCausedByMissingKeyError(filter: Filter): Throwable = {
+ new AnalysisException(
+ s"Fail to rebuild expression: missing key $filter in `translatedFilterToExpr`")
+ }
+
+ def dataTypeUnsupportedByDataSourceError(format: String, field: StructField): Throwable = {
+ new AnalysisException(
+ s"$format data source does not support ${field.dataType.catalogString} data type.")
+ }
+
+ def failToResolveDataSourceForTableError(table: CatalogTable, key: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Fail to resolve data source for the table ${table.identifier} since the table
+ |serde property has the duplicated key $key with extra options specified for this
+ |scan operation. To fix this, you can rollback to the legacy behavior of ignoring
+ |the extra options by setting the config
+ |${SQLConf.LEGACY_EXTRA_OPTIONS_BEHAVIOR.key} to `false`, or address the
+ |conflicts of the same config.
+ """.stripMargin)
+ }
+
+ def insertIntoExistingOutputPathError(outputPath: Path): Throwable = {
+ new AnalysisException(s"path $outputPath already exists.")
+ }
+
+ def cannotUseDataTypeForPartitionColumnError(field: StructField): Throwable = {
+ new AnalysisException(s"Cannot use ${field.dataType} for partition column")
+ }
+
+ def cannotUseAllColumnsForPartitionColumnsError(): Throwable = {
+ new AnalysisException(s"Cannot use all columns for partition columns")
+ }
+
+ def partitionColumnNotFoundInSchemaError(col: String, schemaCatalog: String): Throwable = {
+ new AnalysisException(s"Partition column `$col` not found in schema $schemaCatalog")
+ }
+
+ def columnNotFoundInSchemaError(
+ col: StructField, tableSchema: Option[StructType]): Throwable = {
+ new AnalysisException(s"""Column "${col.name}" not found in schema $tableSchema""")
+ }
+
+ def unsupportedDataSourceTypeForDirectQueryOnFilesError(className: String): Throwable = {
+ new AnalysisException(s"Unsupported data source type for direct query on files: $className")
+ }
+
+ def saveDataIntoViewNotAllowedError(): Throwable = {
+ new AnalysisException("Saving data into a view is not allowed.")
+ }
+
+ def tableFormatMisMatchSpecifyOneError(
+ tableName: String, existingProvider: Class[_], specifiedProvider: Class[_]): Throwable = {
+ new AnalysisException(
+ s"""
+ |The format of the existing table $tableName is `${existingProvider.getSimpleName}`.
+ |It doesn't match the specified format `${specifiedProvider.getSimpleName}`.
+ """.stripMargin)
+ }
+
+ def tableLocationMisMatchSpecifyOneError(
+ identifier: TableIdentifier,
+ existingTable: CatalogTable,
+ tableDesc: CatalogTable): Throwable = {
+ new AnalysisException(
+ s"""
+ |The location of the existing table ${identifier.quotedString} is
+ |`${existingTable.location}`. It doesn't match the specified location
+ |`${tableDesc.location}`.
+ """.stripMargin)
+ }
+
+ def tableColumnNumberMisMatchSpecifyOneError(
+ tableName: String,
+ existingTable: CatalogTable,
+ query: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"""
+ |The column number of the existing table $tableName
+ |(${existingTable.schema.catalogString}) doesn't match the data schema
+ |(${query.schema.catalogString})
+ """.stripMargin)
+ }
+
+ def cannotResolveColumnGivenInputColumnsError(col: String, inputColumns: String): Throwable = {
+ new AnalysisException(s"cannot resolve '$col' given input columns: [$inputColumns]")
+ }
+
+ def tablePartitionMisMatchSpecifyOneError(
Review comment:
mismatchedTablePartitionColumnError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
+ new AnalysisException("Can only write data to relations with a single path.")
+ }
+
+ def failedRebuildExpressionCausedByMissingKeyError(filter: Filter): Throwable = {
+ new AnalysisException(
+ s"Fail to rebuild expression: missing key $filter in `translatedFilterToExpr`")
+ }
+
+ def dataTypeUnsupportedByDataSourceError(format: String, field: StructField): Throwable = {
+ new AnalysisException(
+ s"$format data source does not support ${field.dataType.catalogString} data type.")
+ }
+
+ def failToResolveDataSourceForTableError(table: CatalogTable, key: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Fail to resolve data source for the table ${table.identifier} since the table
+ |serde property has the duplicated key $key with extra options specified for this
+ |scan operation. To fix this, you can rollback to the legacy behavior of ignoring
+ |the extra options by setting the config
+ |${SQLConf.LEGACY_EXTRA_OPTIONS_BEHAVIOR.key} to `false`, or address the
+ |conflicts of the same config.
+ """.stripMargin)
+ }
+
+ def insertIntoExistingOutputPathError(outputPath: Path): Throwable = {
+ new AnalysisException(s"path $outputPath already exists.")
+ }
+
+ def cannotUseDataTypeForPartitionColumnError(field: StructField): Throwable = {
+ new AnalysisException(s"Cannot use ${field.dataType} for partition column")
+ }
+
+ def cannotUseAllColumnsForPartitionColumnsError(): Throwable = {
+ new AnalysisException(s"Cannot use all columns for partition columns")
+ }
+
+ def partitionColumnNotFoundInSchemaError(col: String, schemaCatalog: String): Throwable = {
+ new AnalysisException(s"Partition column `$col` not found in schema $schemaCatalog")
+ }
+
+ def columnNotFoundInSchemaError(
+ col: StructField, tableSchema: Option[StructType]): Throwable = {
+ new AnalysisException(s"""Column "${col.name}" not found in schema $tableSchema""")
+ }
+
+ def unsupportedDataSourceTypeForDirectQueryOnFilesError(className: String): Throwable = {
+ new AnalysisException(s"Unsupported data source type for direct query on files: $className")
+ }
+
+ def saveDataIntoViewNotAllowedError(): Throwable = {
+ new AnalysisException("Saving data into a view is not allowed.")
+ }
+
+ def tableFormatMisMatchSpecifyOneError(
+ tableName: String, existingProvider: Class[_], specifiedProvider: Class[_]): Throwable = {
+ new AnalysisException(
+ s"""
+ |The format of the existing table $tableName is `${existingProvider.getSimpleName}`.
+ |It doesn't match the specified format `${specifiedProvider.getSimpleName}`.
+ """.stripMargin)
+ }
+
+ def tableLocationMisMatchSpecifyOneError(
+ identifier: TableIdentifier,
+ existingTable: CatalogTable,
+ tableDesc: CatalogTable): Throwable = {
+ new AnalysisException(
+ s"""
+ |The location of the existing table ${identifier.quotedString} is
+ |`${existingTable.location}`. It doesn't match the specified location
+ |`${tableDesc.location}`.
+ """.stripMargin)
+ }
+
+ def tableColumnNumberMisMatchSpecifyOneError(
Review comment:
mismatchedTableColumnNumberError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
+ new AnalysisException("Can only write data to relations with a single path.")
+ }
+
+ def failedRebuildExpressionCausedByMissingKeyError(filter: Filter): Throwable = {
+ new AnalysisException(
+ s"Fail to rebuild expression: missing key $filter in `translatedFilterToExpr`")
+ }
+
+ def dataTypeUnsupportedByDataSourceError(format: String, field: StructField): Throwable = {
+ new AnalysisException(
+ s"$format data source does not support ${field.dataType.catalogString} data type.")
+ }
+
+ def failToResolveDataSourceForTableError(table: CatalogTable, key: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Fail to resolve data source for the table ${table.identifier} since the table
+ |serde property has the duplicated key $key with extra options specified for this
+ |scan operation. To fix this, you can rollback to the legacy behavior of ignoring
+ |the extra options by setting the config
+ |${SQLConf.LEGACY_EXTRA_OPTIONS_BEHAVIOR.key} to `false`, or address the
+ |conflicts of the same config.
+ """.stripMargin)
+ }
+
+ def insertIntoExistingOutputPathError(outputPath: Path): Throwable = {
+ new AnalysisException(s"path $outputPath already exists.")
+ }
+
+ def cannotUseDataTypeForPartitionColumnError(field: StructField): Throwable = {
+ new AnalysisException(s"Cannot use ${field.dataType} for partition column")
+ }
+
+ def cannotUseAllColumnsForPartitionColumnsError(): Throwable = {
+ new AnalysisException(s"Cannot use all columns for partition columns")
+ }
+
+ def partitionColumnNotFoundInSchemaError(col: String, schemaCatalog: String): Throwable = {
+ new AnalysisException(s"Partition column `$col` not found in schema $schemaCatalog")
+ }
+
+ def columnNotFoundInSchemaError(
+ col: StructField, tableSchema: Option[StructType]): Throwable = {
+ new AnalysisException(s"""Column "${col.name}" not found in schema $tableSchema""")
+ }
+
+ def unsupportedDataSourceTypeForDirectQueryOnFilesError(className: String): Throwable = {
+ new AnalysisException(s"Unsupported data source type for direct query on files: $className")
+ }
+
+ def saveDataIntoViewNotAllowedError(): Throwable = {
+ new AnalysisException("Saving data into a view is not allowed.")
+ }
+
+ def tableFormatMisMatchSpecifyOneError(
Review comment:
mismatchedTableFormatError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
+ new AnalysisException("Can only write data to relations with a single path.")
+ }
+
+ def failedRebuildExpressionCausedByMissingKeyError(filter: Filter): Throwable = {
+ new AnalysisException(
+ s"Fail to rebuild expression: missing key $filter in `translatedFilterToExpr`")
+ }
+
+ def dataTypeUnsupportedByDataSourceError(format: String, field: StructField): Throwable = {
+ new AnalysisException(
+ s"$format data source does not support ${field.dataType.catalogString} data type.")
+ }
+
+ def failToResolveDataSourceForTableError(table: CatalogTable, key: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Fail to resolve data source for the table ${table.identifier} since the table
+ |serde property has the duplicated key $key with extra options specified for this
+ |scan operation. To fix this, you can rollback to the legacy behavior of ignoring
+ |the extra options by setting the config
+ |${SQLConf.LEGACY_EXTRA_OPTIONS_BEHAVIOR.key} to `false`, or address the
+ |conflicts of the same config.
+ """.stripMargin)
+ }
+
+ def insertIntoExistingOutputPathError(outputPath: Path): Throwable = {
+ new AnalysisException(s"path $outputPath already exists.")
+ }
+
+ def cannotUseDataTypeForPartitionColumnError(field: StructField): Throwable = {
+ new AnalysisException(s"Cannot use ${field.dataType} for partition column")
+ }
+
+ def cannotUseAllColumnsForPartitionColumnsError(): Throwable = {
+ new AnalysisException(s"Cannot use all columns for partition columns")
+ }
+
+ def partitionColumnNotFoundInSchemaError(col: String, schemaCatalog: String): Throwable = {
+ new AnalysisException(s"Partition column `$col` not found in schema $schemaCatalog")
+ }
+
+ def columnNotFoundInSchemaError(
+ col: StructField, tableSchema: Option[StructType]): Throwable = {
+ new AnalysisException(s"""Column "${col.name}" not found in schema $tableSchema""")
+ }
+
+ def unsupportedDataSourceTypeForDirectQueryOnFilesError(className: String): Throwable = {
+ new AnalysisException(s"Unsupported data source type for direct query on files: $className")
+ }
+
+ def saveDataIntoViewNotAllowedError(): Throwable = {
+ new AnalysisException("Saving data into a view is not allowed.")
+ }
+
+ def tableFormatMisMatchSpecifyOneError(
+ tableName: String, existingProvider: Class[_], specifiedProvider: Class[_]): Throwable = {
+ new AnalysisException(
+ s"""
+ |The format of the existing table $tableName is `${existingProvider.getSimpleName}`.
+ |It doesn't match the specified format `${specifiedProvider.getSimpleName}`.
+ """.stripMargin)
+ }
+
+ def tableLocationMisMatchSpecifyOneError(
+ identifier: TableIdentifier,
+ existingTable: CatalogTable,
+ tableDesc: CatalogTable): Throwable = {
+ new AnalysisException(
+ s"""
+ |The location of the existing table ${identifier.quotedString} is
+ |`${existingTable.location}`. It doesn't match the specified location
+ |`${tableDesc.location}`.
+ """.stripMargin)
+ }
+
+ def tableColumnNumberMisMatchSpecifyOneError(
+ tableName: String,
+ existingTable: CatalogTable,
+ query: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"""
+ |The column number of the existing table $tableName
+ |(${existingTable.schema.catalogString}) doesn't match the data schema
+ |(${query.schema.catalogString})
+ """.stripMargin)
+ }
+
+ def cannotResolveColumnGivenInputColumnsError(col: String, inputColumns: String): Throwable = {
+ new AnalysisException(s"cannot resolve '$col' given input columns: [$inputColumns]")
+ }
+
+ def tablePartitionMisMatchSpecifyOneError(
+ tableName: String,
+ specifiedPartCols: Seq[String],
+ existingPartCols: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Specified partitioning does not match that of the existing table $tableName.
+ |Specified partition columns: [${specifiedPartCols.mkString(", ")}]
+ |Existing partition columns: [$existingPartCols]
+ """.stripMargin)
+ }
+
+ def tableBucketMisMatchSpecifyOneError(
+ tableName: String,
+ specifiedBucketString: String,
+ existingBucketString: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Specified bucketing does not match that of the existing table $tableName.
+ |Specified bucketing: $specifiedBucketString
+ |Existing bucketing: $existingBucketString
+ """.stripMargin)
+ }
+
+ def specifyPartitionNotAllowedWhenTableSchemaNotDefinedError(): Throwable = {
+ new AnalysisException("It is not allowed to specify partitioning when the " +
+ "table schema is not defined.")
+ }
+
+ def bucketColumnIsPartOfPartitionColumnsNotAllowedError(
Review comment:
bucketingColumnCannotBePartOfPartitionColumnsError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
+ new AnalysisException("Can only write data to relations with a single path.")
+ }
+
+ def failedRebuildExpressionCausedByMissingKeyError(filter: Filter): Throwable = {
+ new AnalysisException(
+ s"Fail to rebuild expression: missing key $filter in `translatedFilterToExpr`")
+ }
+
+ def dataTypeUnsupportedByDataSourceError(format: String, field: StructField): Throwable = {
+ new AnalysisException(
+ s"$format data source does not support ${field.dataType.catalogString} data type.")
+ }
+
+ def failToResolveDataSourceForTableError(table: CatalogTable, key: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Fail to resolve data source for the table ${table.identifier} since the table
+ |serde property has the duplicated key $key with extra options specified for this
+ |scan operation. To fix this, you can rollback to the legacy behavior of ignoring
+ |the extra options by setting the config
+ |${SQLConf.LEGACY_EXTRA_OPTIONS_BEHAVIOR.key} to `false`, or address the
+ |conflicts of the same config.
+ """.stripMargin)
+ }
+
+ def insertIntoExistingOutputPathError(outputPath: Path): Throwable = {
+ new AnalysisException(s"path $outputPath already exists.")
+ }
+
+ def cannotUseDataTypeForPartitionColumnError(field: StructField): Throwable = {
+ new AnalysisException(s"Cannot use ${field.dataType} for partition column")
+ }
+
+ def cannotUseAllColumnsForPartitionColumnsError(): Throwable = {
+ new AnalysisException(s"Cannot use all columns for partition columns")
+ }
+
+ def partitionColumnNotFoundInSchemaError(col: String, schemaCatalog: String): Throwable = {
+ new AnalysisException(s"Partition column `$col` not found in schema $schemaCatalog")
+ }
+
+ def columnNotFoundInSchemaError(
+ col: StructField, tableSchema: Option[StructType]): Throwable = {
+ new AnalysisException(s"""Column "${col.name}" not found in schema $tableSchema""")
+ }
+
+ def unsupportedDataSourceTypeForDirectQueryOnFilesError(className: String): Throwable = {
+ new AnalysisException(s"Unsupported data source type for direct query on files: $className")
+ }
+
+ def saveDataIntoViewNotAllowedError(): Throwable = {
+ new AnalysisException("Saving data into a view is not allowed.")
+ }
+
+ def tableFormatMisMatchSpecifyOneError(
+ tableName: String, existingProvider: Class[_], specifiedProvider: Class[_]): Throwable = {
+ new AnalysisException(
+ s"""
+ |The format of the existing table $tableName is `${existingProvider.getSimpleName}`.
+ |It doesn't match the specified format `${specifiedProvider.getSimpleName}`.
+ """.stripMargin)
+ }
+
+ def tableLocationMisMatchSpecifyOneError(
Review comment:
mismatchedTableLocationError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
+ new AnalysisException("Can only write data to relations with a single path.")
+ }
+
+ def failedRebuildExpressionCausedByMissingKeyError(filter: Filter): Throwable = {
+ new AnalysisException(
+ s"Fail to rebuild expression: missing key $filter in `translatedFilterToExpr`")
+ }
+
+ def dataTypeUnsupportedByDataSourceError(format: String, field: StructField): Throwable = {
+ new AnalysisException(
+ s"$format data source does not support ${field.dataType.catalogString} data type.")
+ }
+
+ def failToResolveDataSourceForTableError(table: CatalogTable, key: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Fail to resolve data source for the table ${table.identifier} since the table
+ |serde property has the duplicated key $key with extra options specified for this
+ |scan operation. To fix this, you can rollback to the legacy behavior of ignoring
+ |the extra options by setting the config
+ |${SQLConf.LEGACY_EXTRA_OPTIONS_BEHAVIOR.key} to `false`, or address the
+ |conflicts of the same config.
+ """.stripMargin)
+ }
+
+ def insertIntoExistingOutputPathError(outputPath: Path): Throwable = {
+ new AnalysisException(s"path $outputPath already exists.")
+ }
+
+ def cannotUseDataTypeForPartitionColumnError(field: StructField): Throwable = {
+ new AnalysisException(s"Cannot use ${field.dataType} for partition column")
+ }
+
+ def cannotUseAllColumnsForPartitionColumnsError(): Throwable = {
+ new AnalysisException(s"Cannot use all columns for partition columns")
+ }
+
+ def partitionColumnNotFoundInSchemaError(col: String, schemaCatalog: String): Throwable = {
+ new AnalysisException(s"Partition column `$col` not found in schema $schemaCatalog")
+ }
+
+ def columnNotFoundInSchemaError(
+ col: StructField, tableSchema: Option[StructType]): Throwable = {
+ new AnalysisException(s"""Column "${col.name}" not found in schema $tableSchema""")
+ }
+
+ def unsupportedDataSourceTypeForDirectQueryOnFilesError(className: String): Throwable = {
+ new AnalysisException(s"Unsupported data source type for direct query on files: $className")
+ }
+
+ def saveDataIntoViewNotAllowedError(): Throwable = {
+ new AnalysisException("Saving data into a view is not allowed.")
+ }
+
+ def tableFormatMisMatchSpecifyOneError(
+ tableName: String, existingProvider: Class[_], specifiedProvider: Class[_]): Throwable = {
+ new AnalysisException(
+ s"""
+ |The format of the existing table $tableName is `${existingProvider.getSimpleName}`.
+ |It doesn't match the specified format `${specifiedProvider.getSimpleName}`.
+ """.stripMargin)
+ }
+
+ def tableLocationMisMatchSpecifyOneError(
+ identifier: TableIdentifier,
+ existingTable: CatalogTable,
+ tableDesc: CatalogTable): Throwable = {
+ new AnalysisException(
+ s"""
+ |The location of the existing table ${identifier.quotedString} is
+ |`${existingTable.location}`. It doesn't match the specified location
+ |`${tableDesc.location}`.
+ """.stripMargin)
+ }
+
+ def tableColumnNumberMisMatchSpecifyOneError(
+ tableName: String,
+ existingTable: CatalogTable,
+ query: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"""
+ |The column number of the existing table $tableName
+ |(${existingTable.schema.catalogString}) doesn't match the data schema
+ |(${query.schema.catalogString})
+ """.stripMargin)
+ }
+
+ def cannotResolveColumnGivenInputColumnsError(col: String, inputColumns: String): Throwable = {
+ new AnalysisException(s"cannot resolve '$col' given input columns: [$inputColumns]")
+ }
+
+ def tablePartitionMisMatchSpecifyOneError(
+ tableName: String,
+ specifiedPartCols: Seq[String],
+ existingPartCols: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Specified partitioning does not match that of the existing table $tableName.
+ |Specified partition columns: [${specifiedPartCols.mkString(", ")}]
+ |Existing partition columns: [$existingPartCols]
+ """.stripMargin)
+ }
+
+ def tableBucketMisMatchSpecifyOneError(
+ tableName: String,
+ specifiedBucketString: String,
+ existingBucketString: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Specified bucketing does not match that of the existing table $tableName.
+ |Specified bucketing: $specifiedBucketString
+ |Existing bucketing: $existingBucketString
+ """.stripMargin)
+ }
+
+ def specifyPartitionNotAllowedWhenTableSchemaNotDefinedError(): Throwable = {
+ new AnalysisException("It is not allowed to specify partitioning when the " +
+ "table schema is not defined.")
+ }
+
+ def bucketColumnIsPartOfPartitionColumnsNotAllowedError(
+ bucketCol: String, normalizedPartCols: Seq[String]): Throwable = {
+ new AnalysisException(s"bucketing column '$bucketCol' should not be part of " +
+ s"partition columns '${normalizedPartCols.mkString(", ")}'")
+ }
+
+ def bucketSortingColumnIsPartOfPartitionColumnsNotAllowedError(
Review comment:
bucketSortingColumnCannotBePartOfPartitionColumnsError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
+ new AnalysisException("Can only write data to relations with a single path.")
+ }
+
+ def failedRebuildExpressionCausedByMissingKeyError(filter: Filter): Throwable = {
+ new AnalysisException(
+ s"Fail to rebuild expression: missing key $filter in `translatedFilterToExpr`")
+ }
+
+ def dataTypeUnsupportedByDataSourceError(format: String, field: StructField): Throwable = {
+ new AnalysisException(
+ s"$format data source does not support ${field.dataType.catalogString} data type.")
+ }
+
+ def failToResolveDataSourceForTableError(table: CatalogTable, key: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Fail to resolve data source for the table ${table.identifier} since the table
+ |serde property has the duplicated key $key with extra options specified for this
+ |scan operation. To fix this, you can rollback to the legacy behavior of ignoring
+ |the extra options by setting the config
+ |${SQLConf.LEGACY_EXTRA_OPTIONS_BEHAVIOR.key} to `false`, or address the
+ |conflicts of the same config.
+ """.stripMargin)
+ }
+
+ def insertIntoExistingOutputPathError(outputPath: Path): Throwable = {
+ new AnalysisException(s"path $outputPath already exists.")
+ }
+
+ def cannotUseDataTypeForPartitionColumnError(field: StructField): Throwable = {
+ new AnalysisException(s"Cannot use ${field.dataType} for partition column")
+ }
+
+ def cannotUseAllColumnsForPartitionColumnsError(): Throwable = {
+ new AnalysisException(s"Cannot use all columns for partition columns")
+ }
+
+ def partitionColumnNotFoundInSchemaError(col: String, schemaCatalog: String): Throwable = {
+ new AnalysisException(s"Partition column `$col` not found in schema $schemaCatalog")
+ }
+
+ def columnNotFoundInSchemaError(
+ col: StructField, tableSchema: Option[StructType]): Throwable = {
+ new AnalysisException(s"""Column "${col.name}" not found in schema $tableSchema""")
+ }
+
+ def unsupportedDataSourceTypeForDirectQueryOnFilesError(className: String): Throwable = {
+ new AnalysisException(s"Unsupported data source type for direct query on files: $className")
+ }
+
+ def saveDataIntoViewNotAllowedError(): Throwable = {
+ new AnalysisException("Saving data into a view is not allowed.")
+ }
+
+ def tableFormatMisMatchSpecifyOneError(
+ tableName: String, existingProvider: Class[_], specifiedProvider: Class[_]): Throwable = {
+ new AnalysisException(
+ s"""
+ |The format of the existing table $tableName is `${existingProvider.getSimpleName}`.
+ |It doesn't match the specified format `${specifiedProvider.getSimpleName}`.
+ """.stripMargin)
+ }
+
+ def tableLocationMisMatchSpecifyOneError(
+ identifier: TableIdentifier,
+ existingTable: CatalogTable,
+ tableDesc: CatalogTable): Throwable = {
+ new AnalysisException(
+ s"""
+ |The location of the existing table ${identifier.quotedString} is
+ |`${existingTable.location}`. It doesn't match the specified location
+ |`${tableDesc.location}`.
+ """.stripMargin)
+ }
+
+ def tableColumnNumberMisMatchSpecifyOneError(
+ tableName: String,
+ existingTable: CatalogTable,
+ query: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"""
+ |The column number of the existing table $tableName
+ |(${existingTable.schema.catalogString}) doesn't match the data schema
+ |(${query.schema.catalogString})
+ """.stripMargin)
+ }
+
+ def cannotResolveColumnGivenInputColumnsError(col: String, inputColumns: String): Throwable = {
+ new AnalysisException(s"cannot resolve '$col' given input columns: [$inputColumns]")
+ }
+
+ def tablePartitionMisMatchSpecifyOneError(
+ tableName: String,
+ specifiedPartCols: Seq[String],
+ existingPartCols: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Specified partitioning does not match that of the existing table $tableName.
+ |Specified partition columns: [${specifiedPartCols.mkString(", ")}]
+ |Existing partition columns: [$existingPartCols]
+ """.stripMargin)
+ }
+
+ def tableBucketMisMatchSpecifyOneError(
+ tableName: String,
+ specifiedBucketString: String,
+ existingBucketString: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Specified bucketing does not match that of the existing table $tableName.
+ |Specified bucketing: $specifiedBucketString
+ |Existing bucketing: $existingBucketString
+ """.stripMargin)
+ }
+
+ def specifyPartitionNotAllowedWhenTableSchemaNotDefinedError(): Throwable = {
+ new AnalysisException("It is not allowed to specify partitioning when the " +
+ "table schema is not defined.")
+ }
+
+ def bucketColumnIsPartOfPartitionColumnsNotAllowedError(
+ bucketCol: String, normalizedPartCols: Seq[String]): Throwable = {
+ new AnalysisException(s"bucketing column '$bucketCol' should not be part of " +
+ s"partition columns '${normalizedPartCols.mkString(", ")}'")
+ }
+
+ def bucketSortingColumnIsPartOfPartitionColumnsNotAllowedError(
+ sortCol: String, normalizedPartCols: Seq[String]): Throwable = {
+ new AnalysisException(s"bucket sorting column '$sortCol' should not be part of " +
+ s"partition columns '${normalizedPartCols.mkString(", ")}'")
+ }
+
+ def columnNumberOfInsertedDataMisMatchTargetTableError(
Review comment:
mismatchedInsertedDataColumnNumberError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
Review comment:
failedToFindDataSourceError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
+ new AnalysisException("Can only write data to relations with a single path.")
+ }
+
+ def failedRebuildExpressionCausedByMissingKeyError(filter: Filter): Throwable = {
+ new AnalysisException(
+ s"Fail to rebuild expression: missing key $filter in `translatedFilterToExpr`")
+ }
+
+ def dataTypeUnsupportedByDataSourceError(format: String, field: StructField): Throwable = {
+ new AnalysisException(
+ s"$format data source does not support ${field.dataType.catalogString} data type.")
+ }
+
+ def failToResolveDataSourceForTableError(table: CatalogTable, key: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Fail to resolve data source for the table ${table.identifier} since the table
+ |serde property has the duplicated key $key with extra options specified for this
+ |scan operation. To fix this, you can rollback to the legacy behavior of ignoring
+ |the extra options by setting the config
+ |${SQLConf.LEGACY_EXTRA_OPTIONS_BEHAVIOR.key} to `false`, or address the
+ |conflicts of the same config.
+ """.stripMargin)
+ }
+
+ def insertIntoExistingOutputPathError(outputPath: Path): Throwable = {
+ new AnalysisException(s"path $outputPath already exists.")
+ }
+
+ def cannotUseDataTypeForPartitionColumnError(field: StructField): Throwable = {
+ new AnalysisException(s"Cannot use ${field.dataType} for partition column")
+ }
+
+ def cannotUseAllColumnsForPartitionColumnsError(): Throwable = {
+ new AnalysisException(s"Cannot use all columns for partition columns")
+ }
+
+ def partitionColumnNotFoundInSchemaError(col: String, schemaCatalog: String): Throwable = {
+ new AnalysisException(s"Partition column `$col` not found in schema $schemaCatalog")
+ }
+
+ def columnNotFoundInSchemaError(
+ col: StructField, tableSchema: Option[StructType]): Throwable = {
+ new AnalysisException(s"""Column "${col.name}" not found in schema $tableSchema""")
+ }
+
+ def unsupportedDataSourceTypeForDirectQueryOnFilesError(className: String): Throwable = {
+ new AnalysisException(s"Unsupported data source type for direct query on files: $className")
+ }
+
+ def saveDataIntoViewNotAllowedError(): Throwable = {
+ new AnalysisException("Saving data into a view is not allowed.")
+ }
+
+ def tableFormatMisMatchSpecifyOneError(
+ tableName: String, existingProvider: Class[_], specifiedProvider: Class[_]): Throwable = {
+ new AnalysisException(
+ s"""
+ |The format of the existing table $tableName is `${existingProvider.getSimpleName}`.
+ |It doesn't match the specified format `${specifiedProvider.getSimpleName}`.
+ """.stripMargin)
+ }
+
+ def tableLocationMisMatchSpecifyOneError(
+ identifier: TableIdentifier,
+ existingTable: CatalogTable,
+ tableDesc: CatalogTable): Throwable = {
+ new AnalysisException(
+ s"""
+ |The location of the existing table ${identifier.quotedString} is
+ |`${existingTable.location}`. It doesn't match the specified location
+ |`${tableDesc.location}`.
+ """.stripMargin)
+ }
+
+ def tableColumnNumberMisMatchSpecifyOneError(
+ tableName: String,
+ existingTable: CatalogTable,
+ query: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"""
+ |The column number of the existing table $tableName
+ |(${existingTable.schema.catalogString}) doesn't match the data schema
+ |(${query.schema.catalogString})
+ """.stripMargin)
+ }
+
+ def cannotResolveColumnGivenInputColumnsError(col: String, inputColumns: String): Throwable = {
+ new AnalysisException(s"cannot resolve '$col' given input columns: [$inputColumns]")
+ }
+
+ def tablePartitionMisMatchSpecifyOneError(
+ tableName: String,
+ specifiedPartCols: Seq[String],
+ existingPartCols: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Specified partitioning does not match that of the existing table $tableName.
+ |Specified partition columns: [${specifiedPartCols.mkString(", ")}]
+ |Existing partition columns: [$existingPartCols]
+ """.stripMargin)
+ }
+
+ def tableBucketMisMatchSpecifyOneError(
+ tableName: String,
+ specifiedBucketString: String,
+ existingBucketString: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Specified bucketing does not match that of the existing table $tableName.
+ |Specified bucketing: $specifiedBucketString
+ |Existing bucketing: $existingBucketString
+ """.stripMargin)
+ }
+
+ def specifyPartitionNotAllowedWhenTableSchemaNotDefinedError(): Throwable = {
+ new AnalysisException("It is not allowed to specify partitioning when the " +
+ "table schema is not defined.")
+ }
+
+ def bucketColumnIsPartOfPartitionColumnsNotAllowedError(
+ bucketCol: String, normalizedPartCols: Seq[String]): Throwable = {
+ new AnalysisException(s"bucketing column '$bucketCol' should not be part of " +
+ s"partition columns '${normalizedPartCols.mkString(", ")}'")
+ }
+
+ def bucketSortingColumnIsPartOfPartitionColumnsNotAllowedError(
+ sortCol: String, normalizedPartCols: Seq[String]): Throwable = {
+ new AnalysisException(s"bucket sorting column '$sortCol' should not be part of " +
+ s"partition columns '${normalizedPartCols.mkString(", ")}'")
+ }
+
+ def columnNumberOfInsertedDataMisMatchTargetTableError(
+ tableName: String, insert: InsertIntoStatement, staticPartCols: Set[String]): Throwable = {
+ new AnalysisException(
+ s"$tableName requires that the data to be inserted have the same number of columns as " +
+ s"the target table: target table has ${insert.table.output.size} column(s) but the " +
+ s"inserted data has ${insert.query.output.length + staticPartCols.size} column(s), " +
+ s"including ${staticPartCols.size} partition column(s) having constant value(s).")
+ }
+
+ def requestedPartitionsMisMatchTablePartitionsError(
Review comment:
requestedPartitionsMismatchTablePartitionsError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
+ new AnalysisException("Can only write data to relations with a single path.")
+ }
+
+ def failedRebuildExpressionCausedByMissingKeyError(filter: Filter): Throwable = {
+ new AnalysisException(
+ s"Fail to rebuild expression: missing key $filter in `translatedFilterToExpr`")
+ }
+
+ def dataTypeUnsupportedByDataSourceError(format: String, field: StructField): Throwable = {
+ new AnalysisException(
+ s"$format data source does not support ${field.dataType.catalogString} data type.")
+ }
+
+ def failToResolveDataSourceForTableError(table: CatalogTable, key: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Fail to resolve data source for the table ${table.identifier} since the table
+ |serde property has the duplicated key $key with extra options specified for this
+ |scan operation. To fix this, you can rollback to the legacy behavior of ignoring
+ |the extra options by setting the config
+ |${SQLConf.LEGACY_EXTRA_OPTIONS_BEHAVIOR.key} to `false`, or address the
+ |conflicts of the same config.
+ """.stripMargin)
+ }
+
+ def insertIntoExistingOutputPathError(outputPath: Path): Throwable = {
+ new AnalysisException(s"path $outputPath already exists.")
+ }
+
+ def cannotUseDataTypeForPartitionColumnError(field: StructField): Throwable = {
+ new AnalysisException(s"Cannot use ${field.dataType} for partition column")
+ }
+
+ def cannotUseAllColumnsForPartitionColumnsError(): Throwable = {
+ new AnalysisException(s"Cannot use all columns for partition columns")
+ }
+
+ def partitionColumnNotFoundInSchemaError(col: String, schemaCatalog: String): Throwable = {
+ new AnalysisException(s"Partition column `$col` not found in schema $schemaCatalog")
+ }
+
+ def columnNotFoundInSchemaError(
+ col: StructField, tableSchema: Option[StructType]): Throwable = {
+ new AnalysisException(s"""Column "${col.name}" not found in schema $tableSchema""")
+ }
+
+ def unsupportedDataSourceTypeForDirectQueryOnFilesError(className: String): Throwable = {
+ new AnalysisException(s"Unsupported data source type for direct query on files: $className")
+ }
+
+ def saveDataIntoViewNotAllowedError(): Throwable = {
+ new AnalysisException("Saving data into a view is not allowed.")
+ }
+
+ def tableFormatMisMatchSpecifyOneError(
+ tableName: String, existingProvider: Class[_], specifiedProvider: Class[_]): Throwable = {
+ new AnalysisException(
+ s"""
+ |The format of the existing table $tableName is `${existingProvider.getSimpleName}`.
+ |It doesn't match the specified format `${specifiedProvider.getSimpleName}`.
+ """.stripMargin)
+ }
+
+ def tableLocationMisMatchSpecifyOneError(
+ identifier: TableIdentifier,
+ existingTable: CatalogTable,
+ tableDesc: CatalogTable): Throwable = {
+ new AnalysisException(
+ s"""
+ |The location of the existing table ${identifier.quotedString} is
+ |`${existingTable.location}`. It doesn't match the specified location
+ |`${tableDesc.location}`.
+ """.stripMargin)
+ }
+
+ def tableColumnNumberMisMatchSpecifyOneError(
+ tableName: String,
+ existingTable: CatalogTable,
+ query: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"""
+ |The column number of the existing table $tableName
+ |(${existingTable.schema.catalogString}) doesn't match the data schema
+ |(${query.schema.catalogString})
+ """.stripMargin)
+ }
+
+ def cannotResolveColumnGivenInputColumnsError(col: String, inputColumns: String): Throwable = {
+ new AnalysisException(s"cannot resolve '$col' given input columns: [$inputColumns]")
+ }
+
+ def tablePartitionMisMatchSpecifyOneError(
+ tableName: String,
+ specifiedPartCols: Seq[String],
+ existingPartCols: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Specified partitioning does not match that of the existing table $tableName.
+ |Specified partition columns: [${specifiedPartCols.mkString(", ")}]
+ |Existing partition columns: [$existingPartCols]
+ """.stripMargin)
+ }
+
+ def tableBucketMisMatchSpecifyOneError(
+ tableName: String,
+ specifiedBucketString: String,
+ existingBucketString: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Specified bucketing does not match that of the existing table $tableName.
+ |Specified bucketing: $specifiedBucketString
+ |Existing bucketing: $existingBucketString
+ """.stripMargin)
+ }
+
+ def specifyPartitionNotAllowedWhenTableSchemaNotDefinedError(): Throwable = {
+ new AnalysisException("It is not allowed to specify partitioning when the " +
+ "table schema is not defined.")
+ }
+
+ def bucketColumnIsPartOfPartitionColumnsNotAllowedError(
+ bucketCol: String, normalizedPartCols: Seq[String]): Throwable = {
+ new AnalysisException(s"bucketing column '$bucketCol' should not be part of " +
+ s"partition columns '${normalizedPartCols.mkString(", ")}'")
+ }
+
+ def bucketSortingColumnIsPartOfPartitionColumnsNotAllowedError(
+ sortCol: String, normalizedPartCols: Seq[String]): Throwable = {
+ new AnalysisException(s"bucket sorting column '$sortCol' should not be part of " +
+ s"partition columns '${normalizedPartCols.mkString(", ")}'")
+ }
+
+ def columnNumberOfInsertedDataMisMatchTargetTableError(
+ tableName: String, insert: InsertIntoStatement, staticPartCols: Set[String]): Throwable = {
+ new AnalysisException(
+ s"$tableName requires that the data to be inserted have the same number of columns as " +
+ s"the target table: target table has ${insert.table.output.size} column(s) but the " +
+ s"inserted data has ${insert.query.output.length + staticPartCols.size} column(s), " +
+ s"including ${staticPartCols.size} partition column(s) having constant value(s).")
+ }
+
+ def requestedPartitionsMisMatchTablePartitionsError(
+ tableName: String,
+ normalizedPartSpec: Map[String, Option[String]],
+ partColNames: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |Requested partitioning does not match the table $tableName:
+ |Requested partitions: ${normalizedPartSpec.keys.mkString(",")}
+ |Table partitions: ${partColNames.mkString(",")}
+ """.stripMargin)
+ }
+
+ def ddlWithoutHiveSupportEnabledError(detail: String): Throwable = {
+ new AnalysisException(s"Hive support is required to $detail")
+ }
+
+ def createTableColumnTypesOptionColumnNotFoundInSchemaError(
+ col: String, schema: StructType): Throwable = {
+ new AnalysisException(
+ s"createTableColumnTypes option column $col not found in schema ${schema.catalogString}")
+ }
+
+ def parquetTypeUnsupportedError(parquetType: String): Throwable = {
+ new AnalysisException(s"Parquet type not supported: $parquetType")
+ }
+
+ def parquetTypeUnsupportedYetError(parquetType: String): Throwable = {
Review comment:
very interesting :)
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
Review comment:
removedClassInSpark2Error
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
Review comment:
incompatibleDataSourceRegisterError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
Review comment:
sparkUpgradeInWritingDatesError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
Review comment:
sparkUpgradeInReadingDatesError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
Review comment:
multiplePathsSpecifiedError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
Review comment:
taskFailedWhileWritingRowsError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
Review comment:
cannotClearOutputDirectoryError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
+ new AnalysisException("Can only write data to relations with a single path.")
+ }
+
+ def failedRebuildExpressionCausedByMissingKeyError(filter: Filter): Throwable = {
+ new AnalysisException(
+ s"Fail to rebuild expression: missing key $filter in `translatedFilterToExpr`")
+ }
+
+ def dataTypeUnsupportedByDataSourceError(format: String, field: StructField): Throwable = {
+ new AnalysisException(
+ s"$format data source does not support ${field.dataType.catalogString} data type.")
+ }
+
+ def failToResolveDataSourceForTableError(table: CatalogTable, key: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Fail to resolve data source for the table ${table.identifier} since the table
+ |serde property has the duplicated key $key with extra options specified for this
+ |scan operation. To fix this, you can rollback to the legacy behavior of ignoring
+ |the extra options by setting the config
+ |${SQLConf.LEGACY_EXTRA_OPTIONS_BEHAVIOR.key} to `false`, or address the
+ |conflicts of the same config.
+ """.stripMargin)
+ }
+
+ def insertIntoExistingOutputPathError(outputPath: Path): Throwable = {
+ new AnalysisException(s"path $outputPath already exists.")
+ }
+
+ def cannotUseDataTypeForPartitionColumnError(field: StructField): Throwable = {
+ new AnalysisException(s"Cannot use ${field.dataType} for partition column")
+ }
+
+ def cannotUseAllColumnsForPartitionColumnsError(): Throwable = {
+ new AnalysisException(s"Cannot use all columns for partition columns")
+ }
+
+ def partitionColumnNotFoundInSchemaError(col: String, schemaCatalog: String): Throwable = {
+ new AnalysisException(s"Partition column `$col` not found in schema $schemaCatalog")
+ }
+
+ def columnNotFoundInSchemaError(
+ col: StructField, tableSchema: Option[StructType]): Throwable = {
+ new AnalysisException(s"""Column "${col.name}" not found in schema $tableSchema""")
+ }
+
+ def unsupportedDataSourceTypeForDirectQueryOnFilesError(className: String): Throwable = {
+ new AnalysisException(s"Unsupported data source type for direct query on files: $className")
+ }
+
+ def saveDataIntoViewNotAllowedError(): Throwable = {
+ new AnalysisException("Saving data into a view is not allowed.")
+ }
+
+ def tableFormatMisMatchSpecifyOneError(
+ tableName: String, existingProvider: Class[_], specifiedProvider: Class[_]): Throwable = {
+ new AnalysisException(
+ s"""
+ |The format of the existing table $tableName is `${existingProvider.getSimpleName}`.
+ |It doesn't match the specified format `${specifiedProvider.getSimpleName}`.
+ """.stripMargin)
+ }
+
+ def tableLocationMisMatchSpecifyOneError(
+ identifier: TableIdentifier,
+ existingTable: CatalogTable,
+ tableDesc: CatalogTable): Throwable = {
+ new AnalysisException(
+ s"""
+ |The location of the existing table ${identifier.quotedString} is
+ |`${existingTable.location}`. It doesn't match the specified location
+ |`${tableDesc.location}`.
+ """.stripMargin)
+ }
+
+ def tableColumnNumberMisMatchSpecifyOneError(
+ tableName: String,
+ existingTable: CatalogTable,
+ query: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"""
+ |The column number of the existing table $tableName
+ |(${existingTable.schema.catalogString}) doesn't match the data schema
+ |(${query.schema.catalogString})
+ """.stripMargin)
+ }
+
+ def cannotResolveColumnGivenInputColumnsError(col: String, inputColumns: String): Throwable = {
+ new AnalysisException(s"cannot resolve '$col' given input columns: [$inputColumns]")
+ }
+
+ def tablePartitionMisMatchSpecifyOneError(
+ tableName: String,
+ specifiedPartCols: Seq[String],
+ existingPartCols: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Specified partitioning does not match that of the existing table $tableName.
+ |Specified partition columns: [${specifiedPartCols.mkString(", ")}]
+ |Existing partition columns: [$existingPartCols]
+ """.stripMargin)
+ }
+
+ def tableBucketMisMatchSpecifyOneError(
+ tableName: String,
+ specifiedBucketString: String,
+ existingBucketString: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Specified bucketing does not match that of the existing table $tableName.
+ |Specified bucketing: $specifiedBucketString
+ |Existing bucketing: $existingBucketString
+ """.stripMargin)
+ }
+
+ def specifyPartitionNotAllowedWhenTableSchemaNotDefinedError(): Throwable = {
+ new AnalysisException("It is not allowed to specify partitioning when the " +
+ "table schema is not defined.")
+ }
+
+ def bucketColumnIsPartOfPartitionColumnsNotAllowedError(
+ bucketCol: String, normalizedPartCols: Seq[String]): Throwable = {
+ new AnalysisException(s"bucketing column '$bucketCol' should not be part of " +
+ s"partition columns '${normalizedPartCols.mkString(", ")}'")
+ }
+
+ def bucketSortingColumnIsPartOfPartitionColumnsNotAllowedError(
+ sortCol: String, normalizedPartCols: Seq[String]): Throwable = {
+ new AnalysisException(s"bucket sorting column '$sortCol' should not be part of " +
+ s"partition columns '${normalizedPartCols.mkString(", ")}'")
+ }
+
+ def columnNumberOfInsertedDataMisMatchTargetTableError(
+ tableName: String, insert: InsertIntoStatement, staticPartCols: Set[String]): Throwable = {
+ new AnalysisException(
+ s"$tableName requires that the data to be inserted have the same number of columns as " +
+ s"the target table: target table has ${insert.table.output.size} column(s) but the " +
+ s"inserted data has ${insert.query.output.length + staticPartCols.size} column(s), " +
+ s"including ${staticPartCols.size} partition column(s) having constant value(s).")
+ }
+
+ def requestedPartitionsMisMatchTablePartitionsError(
+ tableName: String,
+ normalizedPartSpec: Map[String, Option[String]],
+ partColNames: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |Requested partitioning does not match the table $tableName:
+ |Requested partitions: ${normalizedPartSpec.keys.mkString(",")}
+ |Table partitions: ${partColNames.mkString(",")}
+ """.stripMargin)
+ }
+
+ def ddlWithoutHiveSupportEnabledError(detail: String): Throwable = {
+ new AnalysisException(s"Hive support is required to $detail")
+ }
+
+ def createTableColumnTypesOptionColumnNotFoundInSchemaError(
+ col: String, schema: StructType): Throwable = {
+ new AnalysisException(
+ s"createTableColumnTypes option column $col not found in schema ${schema.catalogString}")
+ }
+
+ def parquetTypeUnsupportedError(parquetType: String): Throwable = {
+ new AnalysisException(s"Parquet type not supported: $parquetType")
+ }
+
+ def parquetTypeUnsupportedYetError(parquetType: String): Throwable = {
+ new AnalysisException(s"Parquet type not yet supported: $parquetType")
+ }
+
+ def illegalParquetTypeError(parquetType: String): Throwable = {
+ new AnalysisException(s"Illegal Parquet type: $parquetType")
+ }
+
+ def unrecognizedParquetTypeError(field: String): Throwable = {
+ new AnalysisException(s"Unrecognized Parquet type: $field")
+ }
+
+ def unsupportedDataTypeError(field: StructField): Throwable = {
Review comment:
cannotConvertDataTypeToParquetTypeError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
+ new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ }
+
+ def unableClearPartitionDirectoryError(path: Path): Throwable = {
+ new IOException(s"Unable to clear partition directory $path prior to writing to it")
+ }
+
+ def failedCastValueToDataTypeForPartitionColumnError(
+ value: String, dataType: DataType, columnName: String): Throwable = {
+ new RuntimeException(s"Failed to cast value `$value` to " +
+ s"`$dataType` for partition column `$columnName`")
+ }
+
+ def typeUnsupportedError(dataType: DataType): Throwable = {
Review comment:
typeUnexpectedError
Note we already have `dataTypeUnexpectedError`, maybe we can add this new error below this error and combine them in the future.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
+ new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ }
+
+ def unableClearPartitionDirectoryError(path: Path): Throwable = {
Review comment:
cannotClearPartitionDirectoryError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
+ new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ }
+
+ def unableClearPartitionDirectoryError(path: Path): Throwable = {
+ new IOException(s"Unable to clear partition directory $path prior to writing to it")
+ }
+
+ def failedCastValueToDataTypeForPartitionColumnError(
+ value: String, dataType: DataType, columnName: String): Throwable = {
+ new RuntimeException(s"Failed to cast value `$value` to " +
+ s"`$dataType` for partition column `$columnName`")
+ }
+
+ def typeUnsupportedError(dataType: DataType): Throwable = {
+ new IllegalArgumentException(s"Unexpected type $dataType")
+ }
+
+ def endOfStreamError(): Throwable = {
+ new java.util.NoSuchElementException("End of stream")
+ }
+
+ def writeUnsupportedForBinaryFileDataSourceError(): Throwable = {
+ new UnsupportedOperationException("Write is not supported for binary file data source")
+ }
+
+ def fileLengthExceedsMaxLengthError(status: FileStatus, maxLength: Int): Throwable = {
+ new SparkException(
+ s"The length of ${status.getPath} is ${status.getLen}, " +
+ s"which exceeds the max length allowed: ${maxLength}.")
+ }
+
+ def unsupportedFieldNameError(fieldName: String): Throwable = {
+ new RuntimeException(s"Unsupported field name: ${fieldName}")
+ }
+
+ def cannotSpecifyBothTableNameAndQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Both '$jdbcTableName' and '$jdbcQueryString' can not be specified at the same time.")
+ }
+
+ def notSpecifyTableNameOrQueryError(
Review comment:
missingJdbcTableNameAndQueryError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
+ new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ }
+
+ def unableClearPartitionDirectoryError(path: Path): Throwable = {
+ new IOException(s"Unable to clear partition directory $path prior to writing to it")
+ }
+
+ def failedCastValueToDataTypeForPartitionColumnError(
+ value: String, dataType: DataType, columnName: String): Throwable = {
+ new RuntimeException(s"Failed to cast value `$value` to " +
+ s"`$dataType` for partition column `$columnName`")
+ }
+
+ def typeUnsupportedError(dataType: DataType): Throwable = {
+ new IllegalArgumentException(s"Unexpected type $dataType")
+ }
+
+ def endOfStreamError(): Throwable = {
+ new java.util.NoSuchElementException("End of stream")
+ }
+
+ def writeUnsupportedForBinaryFileDataSourceError(): Throwable = {
+ new UnsupportedOperationException("Write is not supported for binary file data source")
+ }
+
+ def fileLengthExceedsMaxLengthError(status: FileStatus, maxLength: Int): Throwable = {
+ new SparkException(
+ s"The length of ${status.getPath} is ${status.getLen}, " +
+ s"which exceeds the max length allowed: ${maxLength}.")
+ }
+
+ def unsupportedFieldNameError(fieldName: String): Throwable = {
+ new RuntimeException(s"Unsupported field name: ${fieldName}")
+ }
+
+ def cannotSpecifyBothTableNameAndQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Both '$jdbcTableName' and '$jdbcQueryString' can not be specified at the same time.")
+ }
+
+ def notSpecifyTableNameOrQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Option '$jdbcTableName' or '$jdbcQueryString' is required."
+ )
+ }
+
+ def optionIsEmptyError(optionName: String): Throwable = {
Review comment:
emptyOptionError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
+ new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ }
+
+ def unableClearPartitionDirectoryError(path: Path): Throwable = {
+ new IOException(s"Unable to clear partition directory $path prior to writing to it")
+ }
+
+ def failedCastValueToDataTypeForPartitionColumnError(
+ value: String, dataType: DataType, columnName: String): Throwable = {
+ new RuntimeException(s"Failed to cast value `$value` to " +
+ s"`$dataType` for partition column `$columnName`")
+ }
+
+ def typeUnsupportedError(dataType: DataType): Throwable = {
+ new IllegalArgumentException(s"Unexpected type $dataType")
+ }
+
+ def endOfStreamError(): Throwable = {
+ new java.util.NoSuchElementException("End of stream")
+ }
+
+ def writeUnsupportedForBinaryFileDataSourceError(): Throwable = {
+ new UnsupportedOperationException("Write is not supported for binary file data source")
+ }
+
+ def fileLengthExceedsMaxLengthError(status: FileStatus, maxLength: Int): Throwable = {
+ new SparkException(
+ s"The length of ${status.getPath} is ${status.getLen}, " +
+ s"which exceeds the max length allowed: ${maxLength}.")
+ }
+
+ def unsupportedFieldNameError(fieldName: String): Throwable = {
+ new RuntimeException(s"Unsupported field name: ${fieldName}")
+ }
+
+ def cannotSpecifyBothTableNameAndQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Both '$jdbcTableName' and '$jdbcQueryString' can not be specified at the same time.")
+ }
+
+ def notSpecifyTableNameOrQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Option '$jdbcTableName' or '$jdbcQueryString' is required."
+ )
+ }
+
+ def optionIsEmptyError(optionName: String): Throwable = {
+ new IllegalArgumentException(s"Option `$optionName` can not be empty.")
+ }
+
+ def invalidJdbcTxnIsolationLevelError(jdbcTxnIsolationLevel: String, value: String): Throwable = {
+ new IllegalArgumentException(
+ s"Invalid value `$value` for parameter `$jdbcTxnIsolationLevel`. This can be " +
+ "`NONE`, `READ_UNCOMMITTED`, `READ_COMMITTED`, `REPEATABLE_READ` or `SERIALIZABLE`.")
+ }
+
+ def cannotGetJdbcTypeError(dt: DataType): Throwable = {
+ new IllegalArgumentException(s"Can't get JDBC type for ${dt.catalogString}")
+ }
+
+ def unrecognizedSqlTypeError(sqlType: Int): Throwable = {
+ new SQLException(s"Unrecognized SQL type $sqlType")
+ }
+
+ def unrecognizedTypeError(content: String): Throwable = {
+ new SQLException(s"Unsupported type $content")
+ }
+
+ def unsupportedArrayElementTypeBasedOnBinaryError(dt: DataType): Throwable = {
+ new IllegalArgumentException(s"Unsupported array element " +
+ s"type ${dt.catalogString} based on binary")
+ }
+
+ def nestedArraysUnsupportedError(): Throwable = {
+ new IllegalArgumentException("Nested arrays unsupported")
+ }
+
+ def cannotTranslateNonNullValueForFieldError(pos: Int): Throwable = {
+ new IllegalArgumentException(s"Can't translate non-null value for field $pos")
+ }
+
+ def invalidValueForJdbcNumPartitionsError(n: Int, jdbcNumPartitions: String): Throwable = {
+ new IllegalArgumentException(
+ s"Invalid value `$n` for parameter `$jdbcNumPartitions` in table writing " +
+ "via JDBC. The minimum value is 1.")
+ }
+
+ def transactionUnsupportedByJdbcServerError(): Throwable = {
+ new SQLFeatureNotSupportedException("The target JDBC server does not support " +
+ "transaction and can only support ALTER TABLE with a single action.")
+ }
+
+ def dataTypeUnsupportedYetError(dataType: DataType): Throwable = {
+ new UnsupportedOperationException(s"$dataType is not supported yet.")
+ }
+
+ def unsupportedOperationForDataTypeError(dataType: DataType): Throwable = {
+ new UnsupportedOperationException(s"DataType: ${dataType.catalogString}")
+ }
+
+ def inputFilterNotFullyConvertibleError(own: String): Throwable = {
Review comment:
own? (Maybe orc?)
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
+ new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ }
+
+ def unableClearPartitionDirectoryError(path: Path): Throwable = {
+ new IOException(s"Unable to clear partition directory $path prior to writing to it")
+ }
+
+ def failedCastValueToDataTypeForPartitionColumnError(
+ value: String, dataType: DataType, columnName: String): Throwable = {
+ new RuntimeException(s"Failed to cast value `$value` to " +
+ s"`$dataType` for partition column `$columnName`")
+ }
+
+ def typeUnsupportedError(dataType: DataType): Throwable = {
+ new IllegalArgumentException(s"Unexpected type $dataType")
+ }
+
+ def endOfStreamError(): Throwable = {
+ new java.util.NoSuchElementException("End of stream")
+ }
+
+ def writeUnsupportedForBinaryFileDataSourceError(): Throwable = {
+ new UnsupportedOperationException("Write is not supported for binary file data source")
+ }
+
+ def fileLengthExceedsMaxLengthError(status: FileStatus, maxLength: Int): Throwable = {
+ new SparkException(
+ s"The length of ${status.getPath} is ${status.getLen}, " +
+ s"which exceeds the max length allowed: ${maxLength}.")
+ }
+
+ def unsupportedFieldNameError(fieldName: String): Throwable = {
+ new RuntimeException(s"Unsupported field name: ${fieldName}")
+ }
+
+ def cannotSpecifyBothTableNameAndQueryError(
Review comment:
cannotSpecifyBothJdbcTableNameAndQueryError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
+ new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ }
+
+ def unableClearPartitionDirectoryError(path: Path): Throwable = {
+ new IOException(s"Unable to clear partition directory $path prior to writing to it")
+ }
+
+ def failedCastValueToDataTypeForPartitionColumnError(
+ value: String, dataType: DataType, columnName: String): Throwable = {
+ new RuntimeException(s"Failed to cast value `$value` to " +
+ s"`$dataType` for partition column `$columnName`")
+ }
+
+ def typeUnsupportedError(dataType: DataType): Throwable = {
+ new IllegalArgumentException(s"Unexpected type $dataType")
+ }
+
+ def endOfStreamError(): Throwable = {
+ new java.util.NoSuchElementException("End of stream")
+ }
+
+ def writeUnsupportedForBinaryFileDataSourceError(): Throwable = {
+ new UnsupportedOperationException("Write is not supported for binary file data source")
+ }
+
+ def fileLengthExceedsMaxLengthError(status: FileStatus, maxLength: Int): Throwable = {
+ new SparkException(
+ s"The length of ${status.getPath} is ${status.getLen}, " +
+ s"which exceeds the max length allowed: ${maxLength}.")
+ }
+
+ def unsupportedFieldNameError(fieldName: String): Throwable = {
+ new RuntimeException(s"Unsupported field name: ${fieldName}")
+ }
+
+ def cannotSpecifyBothTableNameAndQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Both '$jdbcTableName' and '$jdbcQueryString' can not be specified at the same time.")
+ }
+
+ def notSpecifyTableNameOrQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Option '$jdbcTableName' or '$jdbcQueryString' is required."
+ )
+ }
+
+ def optionIsEmptyError(optionName: String): Throwable = {
+ new IllegalArgumentException(s"Option `$optionName` can not be empty.")
+ }
+
+ def invalidJdbcTxnIsolationLevelError(jdbcTxnIsolationLevel: String, value: String): Throwable = {
+ new IllegalArgumentException(
+ s"Invalid value `$value` for parameter `$jdbcTxnIsolationLevel`. This can be " +
+ "`NONE`, `READ_UNCOMMITTED`, `READ_COMMITTED`, `REPEATABLE_READ` or `SERIALIZABLE`.")
+ }
+
+ def cannotGetJdbcTypeError(dt: DataType): Throwable = {
+ new IllegalArgumentException(s"Can't get JDBC type for ${dt.catalogString}")
+ }
+
+ def unrecognizedSqlTypeError(sqlType: Int): Throwable = {
+ new SQLException(s"Unrecognized SQL type $sqlType")
+ }
+
+ def unrecognizedTypeError(content: String): Throwable = {
+ new SQLException(s"Unsupported type $content")
+ }
+
+ def unsupportedArrayElementTypeBasedOnBinaryError(dt: DataType): Throwable = {
+ new IllegalArgumentException(s"Unsupported array element " +
+ s"type ${dt.catalogString} based on binary")
+ }
+
+ def nestedArraysUnsupportedError(): Throwable = {
+ new IllegalArgumentException("Nested arrays unsupported")
+ }
+
+ def cannotTranslateNonNullValueForFieldError(pos: Int): Throwable = {
+ new IllegalArgumentException(s"Can't translate non-null value for field $pos")
+ }
+
+ def invalidValueForJdbcNumPartitionsError(n: Int, jdbcNumPartitions: String): Throwable = {
Review comment:
invalidJdbcNumPartitionsError
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
+ new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ }
+
+ def unableClearPartitionDirectoryError(path: Path): Throwable = {
+ new IOException(s"Unable to clear partition directory $path prior to writing to it")
+ }
+
+ def failedCastValueToDataTypeForPartitionColumnError(
+ value: String, dataType: DataType, columnName: String): Throwable = {
+ new RuntimeException(s"Failed to cast value `$value` to " +
+ s"`$dataType` for partition column `$columnName`")
+ }
+
+ def typeUnsupportedError(dataType: DataType): Throwable = {
+ new IllegalArgumentException(s"Unexpected type $dataType")
+ }
+
+ def endOfStreamError(): Throwable = {
+ new java.util.NoSuchElementException("End of stream")
+ }
+
+ def writeUnsupportedForBinaryFileDataSourceError(): Throwable = {
+ new UnsupportedOperationException("Write is not supported for binary file data source")
+ }
+
+ def fileLengthExceedsMaxLengthError(status: FileStatus, maxLength: Int): Throwable = {
+ new SparkException(
+ s"The length of ${status.getPath} is ${status.getLen}, " +
+ s"which exceeds the max length allowed: ${maxLength}.")
+ }
+
+ def unsupportedFieldNameError(fieldName: String): Throwable = {
+ new RuntimeException(s"Unsupported field name: ${fieldName}")
+ }
+
+ def cannotSpecifyBothTableNameAndQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Both '$jdbcTableName' and '$jdbcQueryString' can not be specified at the same time.")
+ }
+
+ def notSpecifyTableNameOrQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Option '$jdbcTableName' or '$jdbcQueryString' is required."
+ )
+ }
+
+ def optionIsEmptyError(optionName: String): Throwable = {
+ new IllegalArgumentException(s"Option `$optionName` can not be empty.")
+ }
+
+ def invalidJdbcTxnIsolationLevelError(jdbcTxnIsolationLevel: String, value: String): Throwable = {
+ new IllegalArgumentException(
+ s"Invalid value `$value` for parameter `$jdbcTxnIsolationLevel`. This can be " +
+ "`NONE`, `READ_UNCOMMITTED`, `READ_COMMITTED`, `REPEATABLE_READ` or `SERIALIZABLE`.")
+ }
+
+ def cannotGetJdbcTypeError(dt: DataType): Throwable = {
+ new IllegalArgumentException(s"Can't get JDBC type for ${dt.catalogString}")
+ }
+
+ def unrecognizedSqlTypeError(sqlType: Int): Throwable = {
+ new SQLException(s"Unrecognized SQL type $sqlType")
+ }
+
+ def unrecognizedTypeError(content: String): Throwable = {
+ new SQLException(s"Unsupported type $content")
+ }
+
+ def unsupportedArrayElementTypeBasedOnBinaryError(dt: DataType): Throwable = {
+ new IllegalArgumentException(s"Unsupported array element " +
+ s"type ${dt.catalogString} based on binary")
+ }
+
+ def nestedArraysUnsupportedError(): Throwable = {
+ new IllegalArgumentException("Nested arrays unsupported")
+ }
+
+ def cannotTranslateNonNullValueForFieldError(pos: Int): Throwable = {
+ new IllegalArgumentException(s"Can't translate non-null value for field $pos")
+ }
+
+ def invalidValueForJdbcNumPartitionsError(n: Int, jdbcNumPartitions: String): Throwable = {
+ new IllegalArgumentException(
+ s"Invalid value `$n` for parameter `$jdbcNumPartitions` in table writing " +
+ "via JDBC. The minimum value is 1.")
+ }
+
+ def transactionUnsupportedByJdbcServerError(): Throwable = {
+ new SQLFeatureNotSupportedException("The target JDBC server does not support " +
+ "transaction and can only support ALTER TABLE with a single action.")
+ }
+
+ def dataTypeUnsupportedYetError(dataType: DataType): Throwable = {
+ new UnsupportedOperationException(s"$dataType is not supported yet.")
+ }
+
+ def unsupportedOperationForDataTypeError(dataType: DataType): Throwable = {
+ new UnsupportedOperationException(s"DataType: ${dataType.catalogString}")
+ }
+
+ def inputFilterNotFullyConvertibleError(own: String): Throwable = {
+ new SparkException(s"The input filter of $own should be fully convertible.")
+ }
+
+ def cannotReadFooterForFileError(file: Path, e: IOException): Throwable = {
+ new SparkException(s"Could not read footer for file: $file", e)
+ }
+
+ def cannotReadFooterForFileError(file: FileStatus, e: RuntimeException): Throwable = {
+ new Exception(s"Could not read footer for file: $file", e)
Review comment:
This one should be IOException
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
+ new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ }
+
+ def unableClearPartitionDirectoryError(path: Path): Throwable = {
+ new IOException(s"Unable to clear partition directory $path prior to writing to it")
+ }
+
+ def failedCastValueToDataTypeForPartitionColumnError(
+ value: String, dataType: DataType, columnName: String): Throwable = {
+ new RuntimeException(s"Failed to cast value `$value` to " +
+ s"`$dataType` for partition column `$columnName`")
+ }
+
+ def typeUnsupportedError(dataType: DataType): Throwable = {
+ new IllegalArgumentException(s"Unexpected type $dataType")
+ }
+
+ def endOfStreamError(): Throwable = {
+ new java.util.NoSuchElementException("End of stream")
+ }
+
+ def writeUnsupportedForBinaryFileDataSourceError(): Throwable = {
+ new UnsupportedOperationException("Write is not supported for binary file data source")
+ }
+
+ def fileLengthExceedsMaxLengthError(status: FileStatus, maxLength: Int): Throwable = {
+ new SparkException(
+ s"The length of ${status.getPath} is ${status.getLen}, " +
+ s"which exceeds the max length allowed: ${maxLength}.")
+ }
+
+ def unsupportedFieldNameError(fieldName: String): Throwable = {
+ new RuntimeException(s"Unsupported field name: ${fieldName}")
+ }
+
+ def cannotSpecifyBothTableNameAndQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Both '$jdbcTableName' and '$jdbcQueryString' can not be specified at the same time.")
+ }
+
+ def notSpecifyTableNameOrQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Option '$jdbcTableName' or '$jdbcQueryString' is required."
+ )
+ }
+
+ def optionIsEmptyError(optionName: String): Throwable = {
+ new IllegalArgumentException(s"Option `$optionName` can not be empty.")
+ }
+
+ def invalidJdbcTxnIsolationLevelError(jdbcTxnIsolationLevel: String, value: String): Throwable = {
+ new IllegalArgumentException(
+ s"Invalid value `$value` for parameter `$jdbcTxnIsolationLevel`. This can be " +
+ "`NONE`, `READ_UNCOMMITTED`, `READ_COMMITTED`, `REPEATABLE_READ` or `SERIALIZABLE`.")
+ }
+
+ def cannotGetJdbcTypeError(dt: DataType): Throwable = {
+ new IllegalArgumentException(s"Can't get JDBC type for ${dt.catalogString}")
+ }
+
+ def unrecognizedSqlTypeError(sqlType: Int): Throwable = {
+ new SQLException(s"Unrecognized SQL type $sqlType")
+ }
+
+ def unrecognizedTypeError(content: String): Throwable = {
Review comment:
unsupportedJdbcTypeError
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #31757:
URL: https://github.com/apache/spark/pull/31757#discussion_r590979859
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
+ new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ }
+
+ def unableClearPartitionDirectoryError(path: Path): Throwable = {
+ new IOException(s"Unable to clear partition directory $path prior to writing to it")
+ }
+
+ def failedCastValueToDataTypeForPartitionColumnError(
+ value: String, dataType: DataType, columnName: String): Throwable = {
+ new RuntimeException(s"Failed to cast value `$value` to " +
+ s"`$dataType` for partition column `$columnName`")
+ }
+
+ def typeUnsupportedError(dataType: DataType): Throwable = {
+ new IllegalArgumentException(s"Unexpected type $dataType")
+ }
+
+ def endOfStreamError(): Throwable = {
+ new java.util.NoSuchElementException("End of stream")
+ }
+
+ def writeUnsupportedForBinaryFileDataSourceError(): Throwable = {
+ new UnsupportedOperationException("Write is not supported for binary file data source")
+ }
+
+ def fileLengthExceedsMaxLengthError(status: FileStatus, maxLength: Int): Throwable = {
+ new SparkException(
+ s"The length of ${status.getPath} is ${status.getLen}, " +
+ s"which exceeds the max length allowed: ${maxLength}.")
+ }
+
+ def unsupportedFieldNameError(fieldName: String): Throwable = {
+ new RuntimeException(s"Unsupported field name: ${fieldName}")
+ }
+
+ def cannotSpecifyBothTableNameAndQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Both '$jdbcTableName' and '$jdbcQueryString' can not be specified at the same time.")
+ }
+
+ def notSpecifyTableNameOrQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Option '$jdbcTableName' or '$jdbcQueryString' is required."
+ )
+ }
+
+ def optionIsEmptyError(optionName: String): Throwable = {
+ new IllegalArgumentException(s"Option `$optionName` can not be empty.")
+ }
+
+ def invalidJdbcTxnIsolationLevelError(jdbcTxnIsolationLevel: String, value: String): Throwable = {
+ new IllegalArgumentException(
+ s"Invalid value `$value` for parameter `$jdbcTxnIsolationLevel`. This can be " +
+ "`NONE`, `READ_UNCOMMITTED`, `READ_COMMITTED`, `REPEATABLE_READ` or `SERIALIZABLE`.")
+ }
+
+ def cannotGetJdbcTypeError(dt: DataType): Throwable = {
+ new IllegalArgumentException(s"Can't get JDBC type for ${dt.catalogString}")
+ }
+
+ def unrecognizedSqlTypeError(sqlType: Int): Throwable = {
+ new SQLException(s"Unrecognized SQL type $sqlType")
+ }
+
+ def unrecognizedTypeError(content: String): Throwable = {
+ new SQLException(s"Unsupported type $content")
+ }
+
+ def unsupportedArrayElementTypeBasedOnBinaryError(dt: DataType): Throwable = {
+ new IllegalArgumentException(s"Unsupported array element " +
+ s"type ${dt.catalogString} based on binary")
+ }
+
+ def nestedArraysUnsupportedError(): Throwable = {
+ new IllegalArgumentException("Nested arrays unsupported")
+ }
+
+ def cannotTranslateNonNullValueForFieldError(pos: Int): Throwable = {
+ new IllegalArgumentException(s"Can't translate non-null value for field $pos")
+ }
+
+ def invalidValueForJdbcNumPartitionsError(n: Int, jdbcNumPartitions: String): Throwable = {
+ new IllegalArgumentException(
+ s"Invalid value `$n` for parameter `$jdbcNumPartitions` in table writing " +
+ "via JDBC. The minimum value is 1.")
+ }
+
+ def transactionUnsupportedByJdbcServerError(): Throwable = {
+ new SQLFeatureNotSupportedException("The target JDBC server does not support " +
+ "transaction and can only support ALTER TABLE with a single action.")
+ }
+
+ def dataTypeUnsupportedYetError(dataType: DataType): Throwable = {
+ new UnsupportedOperationException(s"$dataType is not supported yet.")
+ }
+
+ def unsupportedOperationForDataTypeError(dataType: DataType): Throwable = {
+ new UnsupportedOperationException(s"DataType: ${dataType.catalogString}")
+ }
+
+ def inputFilterNotFullyConvertibleError(own: String): Throwable = {
Review comment:
owner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-791508424
**[Test build #135810 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135810/testReport)** for PR 31757 at commit [`47bc26b`](https://github.com/apache/spark/commit/47bc26bcec5173a1ca5493492848eac7e71374f9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-796406347
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40540/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-791358955
**[Test build #135810 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135810/testReport)** for PR 31757 at commit [`47bc26b`](https://github.com/apache/spark/commit/47bc26bcec5173a1ca5493492848eac7e71374f9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-796379768
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-791311351
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135806/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-800917925
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40736/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #31757:
URL: https://github.com/apache/spark/pull/31757#discussion_r595801953
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -199,7 +187,7 @@ object DataSourceAnalysis extends Rule[LogicalPlan] with CastSupport {
// Sanity check
if (t.location.rootPaths.size != 1) {
- throw new AnalysisException("Can only write data to relations with a single path.")
Review comment:
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-791335790
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-794954444
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40502/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-796523615
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135956/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-791511032
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135810/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-794949016
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40499/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-794833385
**[Test build #135916 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135916/testReport)** for PR 31757 at commit [`668e007`](https://github.com/apache/spark/commit/668e007ad298842eb72ed96aa1e90eed77e5dba3).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-791370974
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40392/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-796523615
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135956/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-791374605
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40392/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-794904507
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40499/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-794832270
**[Test build #135916 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135916/testReport)** for PR 31757 at commit [`668e007`](https://github.com/apache/spark/commit/668e007ad298842eb72ed96aa1e90eed77e5dba3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-791311351
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-796404137
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40540/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-795007230
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40502/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-794847671
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135916/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-796392735
**[Test build #135956 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135956/testReport)** for PR 31757 at commit [`b575885`](https://github.com/apache/spark/commit/b575885942b813eabb188a9b3646ffb652e9bc87).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-794847671
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135916/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-796409433
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40540/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31757:
URL: https://github.com/apache/spark/pull/31757
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-794933085
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40499/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-796392735
**[Test build #135956 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135956/testReport)** for PR 31757 at commit [`b575885`](https://github.com/apache/spark/commit/b575885942b813eabb188a9b3646ffb652e9bc87).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-796409433
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40540/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #31757:
URL: https://github.com/apache/spark/pull/31757#discussion_r590966655
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
Review comment:
I doubt this change. It seems we should preserve it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-797184644
ping @allisonwang-db
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-791321254
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40389/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-801107478
the scala 2.13 failure is unrelated, I'm merging to master, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-800916712
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136155/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-794911909
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135919/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-794980163
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40502/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-800917925
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40736/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-794949016
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40499/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-794832270
**[Test build #135916 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135916/testReport)** for PR 31757 at commit [`668e007`](https://github.com/apache/spark/commit/668e007ad298842eb72ed96aa1e90eed77e5dba3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31757:
URL: https://github.com/apache/spark/pull/31757#discussion_r595780432
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -199,7 +187,7 @@ object DataSourceAnalysis extends Rule[LogicalPlan] with CastSupport {
// Sanity check
if (t.location.rootPaths.size != 1) {
- throw new AnalysisException("Can only write data to relations with a single path.")
Review comment:
The original error message is `Can only write data to relations with a single path`.
We should rename `cannotWriteDataToRelationWithSinglePathError` to `canOnlyWriteDataToRelationWithSinglePathError `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-794911909
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135919/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-795007230
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40502/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #31757:
URL: https://github.com/apache/spark/pull/31757#discussion_r590977493
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
+ new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ }
+
+ def unableClearPartitionDirectoryError(path: Path): Throwable = {
+ new IOException(s"Unable to clear partition directory $path prior to writing to it")
+ }
+
+ def failedCastValueToDataTypeForPartitionColumnError(
+ value: String, dataType: DataType, columnName: String): Throwable = {
+ new RuntimeException(s"Failed to cast value `$value` to " +
+ s"`$dataType` for partition column `$columnName`")
+ }
+
+ def typeUnsupportedError(dataType: DataType): Throwable = {
Review comment:
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #31757:
URL: https://github.com/apache/spark/pull/31757#discussion_r590966655
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -754,4 +756,335 @@ private[spark] object QueryCompilationErrors {
new AnalysisException(s"Invalid partitionExprs specified: $sortOrders For range " +
"partitioning use REPARTITION_BY_RANGE instead.")
}
+
+ def partitionColumnNotSpecifiedError(format: String, partitionColumn: String): Throwable = {
+ new AnalysisException(s"Failed to resolve the schema for $format for " +
+ s"the partition column: $partitionColumn. It must be specified manually.")
+ }
+
+ def dataSchemaNotSpecifiedError(format: String): Throwable = {
+ new AnalysisException(s"Unable to infer schema for $format. It must be specified manually.")
+ }
+
+ def dataPathNotExistError(path: String): Throwable = {
+ new AnalysisException(s"Path does not exist: $path")
+ }
+
+ def outputModeUnsupportedByDataSourceError(
+ className: String, outputMode: OutputMode): Throwable = {
+ new AnalysisException(s"Data source $className does not support $outputMode output mode")
+ }
+
+ def notSpecifySchemaForSchemaRelationProviderError(className: String): Throwable = {
+ new AnalysisException(s"A schema needs to be specified when using $className.")
+ }
+
+ def userSpecifiedSchemaMismatchActualSchemaError(
+ schema: StructType, actualSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The user-specified schema doesn't match the actual schema:
+ |user-specified: ${schema.toDDL}, actual: ${actualSchema.toDDL}. If you're using
+ |DataFrameReader.schema API or creating a table, please do not specify the schema.
+ |Or if you're scanning an existed table, please drop it and re-create it.
+ """.stripMargin)
+ }
+
+ def dataSchemaNotSpecifiedError(format: String, fileCatalog: String): Throwable = {
+ new AnalysisException(
+ s"Unable to infer schema for $format at $fileCatalog. It must be specified manually")
+ }
+
+ def invalidDataSourceError(className: String): Throwable = {
+ new AnalysisException(s"$className is not a valid Spark SQL Data Source.")
+ }
+
+ def cannotSaveIntervalIntoExternalStorageError(): Throwable = {
+ new AnalysisException("Cannot save interval data type into external storage.")
+ }
+
+ def unableResolveAttributeError(name: String, data: LogicalPlan): Throwable = {
+ new AnalysisException(
+ s"Unable to resolve $name given [${data.output.map(_.name).mkString(", ")}]")
+ }
+
+ def orcNotUsedWithHiveEnabledError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Hive built-in ORC data source must be used with Hive support enabled.
+ |Please use the native ORC data source by setting 'spark.sql.orc.impl' to 'native'
+ """.stripMargin)
+ }
+
+ def failedFindAvroDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Avro is built-in but external data
+ |source module since Spark 2.4. Please deploy the application as per
+ |the deployment section of "Apache Avro Data Source Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def failedFindKafkaDataSourceError(provider: String): Throwable = {
+ new AnalysisException(
+ s"""
+ |Failed to find data source: $provider. Please deploy the application as
+ |per the deployment section of "Structured Streaming + Kafka Integration Guide".
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def findMultipleDataSourceError(provider: String, sourceNames: Seq[String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Multiple sources found for $provider (${sourceNames.mkString(", ")}),
+ | please specify the fully qualified class name.
+ """.stripMargin)
+ }
+
+ def writeEmptySchemasUnsupportedByDataSourceError(): Throwable = {
+ new AnalysisException(
+ s"""
+ |Datasource does not support writing empty or nested empty schemas.
+ |Please make sure the data schema has at least one or more column(s).
+ """.stripMargin)
+ }
+
+ def insertMisMatchedColumnNumberError(
+ targetAttributes: Seq[Attribute],
+ sourceAttributes: Seq[Attribute],
+ staticPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of columns as the
+ |target table: target table has ${targetAttributes.size} column(s) but the
+ |inserted data has ${sourceAttributes.size + staticPartitionsSize} column(s),
+ |which contain $staticPartitionsSize partition column(s) having assigned
+ |constant values.
+ """.stripMargin)
+ }
+
+ def insertMisMatchedPartitionNumberError(
+ targetPartitionSchema: StructType,
+ providedPartitionsSize: Int): Throwable = {
+ new AnalysisException(
+ s"""
+ |The data to be inserted needs to have the same number of partition columns
+ |as the target table: target table has ${targetPartitionSchema.fields.size}
+ |partition column(s) but the inserted data has $providedPartitionsSize
+ |partition columns specified.
+ """.stripMargin.replaceAll("\n", " "))
+ }
+
+ def invalidPartitionColumnError(
+ partKey: String, targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |$partKey is not a partition column. Partition columns are
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ """.stripMargin)
+ }
+
+ def specifyMultipleValuesForPartitionColumnError(
+ field: StructField, potentialSpecs: Map[String, String]): Throwable = {
+ new AnalysisException(
+ s"""
+ |Partition column ${field.name} have multiple values specified,
+ |${potentialSpecs.mkString("[", ", ", "]")}. Please only specify a single value.
+ """.stripMargin)
+ }
+
+ def invalidPartitionColumnHavingConstantValueError(
+ targetPartitionSchema: StructType): Throwable = {
+ new AnalysisException(
+ s"""
+ |The ordering of partition columns is
+ |${targetPartitionSchema.fields.map(_.name).mkString("[", ",", "]")}
+ |All partition columns having constant values need to appear before other
+ |partition columns that do not have an assigned constant value.
+ """.stripMargin)
+ }
+
+ def cannotWriteDataToRelationWithSinglePathError(): Throwable = {
Review comment:
I doubt this change. It seems we should preserve it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-792413105
cc @allisonwang-db
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
beliefer commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-801561202
@allisonwang-db Thanks for your review. @cloud-fan Thanks for your work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-791373718
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40392/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #31757:
URL: https://github.com/apache/spark/pull/31757#discussion_r590982213
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
##########
@@ -322,4 +323,237 @@ object QueryExecutionErrors {
def compilerError(e: CompileException): Throwable = {
new CompileException(failedToCompileMsg(e), e.getLocation)
}
+
+ def dataPathNotSpecifiedError(): Throwable = {
+ new IllegalArgumentException("'path' is not specified")
+ }
+
+ def createStreamingSourceNotSpecifySchemaError(): Throwable = {
+ new IllegalArgumentException(
+ s"""
+ |Schema must be specified when creating a streaming source DataFrame. If some
+ |files already exist in the directory, then depending on the file format you
+ |may be able to create a static DataFrame on that directory with
+ |'spark.read.load(directory)' and infer schema from it.
+ """.stripMargin)
+ }
+
+ def streamedOperatorUnsupportedByDataSourceError(
+ className: String, operator: String): Throwable = {
+ new UnsupportedOperationException(
+ s"Data source $className does not support streamed $operator")
+ }
+
+ def allPathsNotExpectedExactlyOneError(allPaths: Seq[String]): Throwable = {
+ new IllegalArgumentException("Expected exactly one path to be specified, but " +
+ s"got: ${allPaths.mkString(", ")}")
+ }
+
+ def failedFindDataSourceError(provider: String, error: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Failed to find data source: $provider. Please find packages at
+ |http://spark.apache.org/third-party-projects.html
+ """.stripMargin, error)
+ }
+
+ def useSpark2RemovedClassesError(className: String, e: Throwable): Throwable = {
+ new ClassNotFoundException(s"$className was removed in Spark 2.0. " +
+ "Please check if your library is compatible with Spark 2.0", e)
+ }
+
+ def findIncompatibleDataSourceRegisterError(e: Throwable): Throwable = {
+ new ClassNotFoundException(
+ s"""
+ |Detected an incompatible DataSourceRegister. Please remove the incompatible
+ |library from classpath or upgrade it. Error: ${e.getMessage}
+ """.stripMargin, e)
+ }
+
+ def unrecognizedFileFormatError(format: String): Throwable = {
+ new IllegalStateException(s"unrecognized format $format")
+ }
+
+ def sparkUpgradeInReadError(
+ format: String, config: String, option: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from $format
+ |files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of
+ |Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set the SQL config
+ |'$config' or the datasource option '$option' to 'LEGACY' to rebase the datetime values
+ |w.r.t. the calendar difference during reading. To read the datetime values as it is,
+ |set the SQL config '$config' or the datasource option '$option' to 'CORRECTED'.
+ """.stripMargin, null)
+ }
+
+ def sparkUpgradeInWriteError(format: String, config: String): SparkUpgradeException = {
+ new SparkUpgradeException("3.0",
+ s"""
+ |writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into $format
+ |files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
+ |later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
+ |Gregorian calendar. See more details in SPARK-31404. You can set $config to 'LEGACY' to
+ |rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
+ |interoperability. Or set $config to 'CORRECTED' to write the datetime values as it is,
+ |if you are 100% sure that the written files will only be read by Spark 3.0+ or other
+ |systems that use Proleptic Gregorian calendar.
+ """.stripMargin, null)
+ }
+
+ def buildReaderUnsupportedForFileFormatError(format: String): Throwable = {
+ new UnsupportedOperationException(s"buildReader is not supported for $format")
+ }
+
+ def jobAbortedError(cause: Throwable): Throwable = {
+ new SparkException("Job aborted.", cause)
+ }
+
+ def taskFailedWhileWriteRowsError(cause: Throwable): Throwable = {
+ new SparkException("Task failed while writing rows.", cause)
+ }
+
+ def readCurrentFileNotFoundError(e: FileNotFoundException): Throwable = {
+ new FileNotFoundException(
+ s"""
+ |${e.getMessage}\n
+ |It is possible the underlying files have been updated. You can explicitly invalidate
+ |the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by
+ |recreating the Dataset/DataFrame involved.
+ """.stripMargin)
+ }
+
+ def unsupportedSaveModeError(saveMode: String, pathExists: Boolean): Throwable = {
+ new IllegalStateException(s"unsupported save mode $saveMode ($pathExists)")
+ }
+
+ def unableClearOutputDirectoryError(staticPrefixPath: Path): Throwable = {
+ new IOException(s"Unable to clear output directory $staticPrefixPath prior to writing to it")
+ }
+
+ def unableClearPartitionDirectoryError(path: Path): Throwable = {
+ new IOException(s"Unable to clear partition directory $path prior to writing to it")
+ }
+
+ def failedCastValueToDataTypeForPartitionColumnError(
+ value: String, dataType: DataType, columnName: String): Throwable = {
+ new RuntimeException(s"Failed to cast value `$value` to " +
+ s"`$dataType` for partition column `$columnName`")
+ }
+
+ def typeUnsupportedError(dataType: DataType): Throwable = {
+ new IllegalArgumentException(s"Unexpected type $dataType")
+ }
+
+ def endOfStreamError(): Throwable = {
+ new java.util.NoSuchElementException("End of stream")
+ }
+
+ def writeUnsupportedForBinaryFileDataSourceError(): Throwable = {
+ new UnsupportedOperationException("Write is not supported for binary file data source")
+ }
+
+ def fileLengthExceedsMaxLengthError(status: FileStatus, maxLength: Int): Throwable = {
+ new SparkException(
+ s"The length of ${status.getPath} is ${status.getLen}, " +
+ s"which exceeds the max length allowed: ${maxLength}.")
+ }
+
+ def unsupportedFieldNameError(fieldName: String): Throwable = {
+ new RuntimeException(s"Unsupported field name: ${fieldName}")
+ }
+
+ def cannotSpecifyBothTableNameAndQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Both '$jdbcTableName' and '$jdbcQueryString' can not be specified at the same time.")
+ }
+
+ def notSpecifyTableNameOrQueryError(
+ jdbcTableName: String, jdbcQueryString: String): Throwable = {
+ new IllegalArgumentException(
+ s"Option '$jdbcTableName' or '$jdbcQueryString' is required."
+ )
+ }
+
+ def optionIsEmptyError(optionName: String): Throwable = {
+ new IllegalArgumentException(s"Option `$optionName` can not be empty.")
+ }
+
+ def invalidJdbcTxnIsolationLevelError(jdbcTxnIsolationLevel: String, value: String): Throwable = {
+ new IllegalArgumentException(
+ s"Invalid value `$value` for parameter `$jdbcTxnIsolationLevel`. This can be " +
+ "`NONE`, `READ_UNCOMMITTED`, `READ_COMMITTED`, `REPEATABLE_READ` or `SERIALIZABLE`.")
+ }
+
+ def cannotGetJdbcTypeError(dt: DataType): Throwable = {
+ new IllegalArgumentException(s"Can't get JDBC type for ${dt.catalogString}")
+ }
+
+ def unrecognizedSqlTypeError(sqlType: Int): Throwable = {
+ new SQLException(s"Unrecognized SQL type $sqlType")
+ }
+
+ def unrecognizedTypeError(content: String): Throwable = {
+ new SQLException(s"Unsupported type $content")
+ }
+
+ def unsupportedArrayElementTypeBasedOnBinaryError(dt: DataType): Throwable = {
+ new IllegalArgumentException(s"Unsupported array element " +
+ s"type ${dt.catalogString} based on binary")
+ }
+
+ def nestedArraysUnsupportedError(): Throwable = {
+ new IllegalArgumentException("Nested arrays unsupported")
+ }
+
+ def cannotTranslateNonNullValueForFieldError(pos: Int): Throwable = {
+ new IllegalArgumentException(s"Can't translate non-null value for field $pos")
+ }
+
+ def invalidValueForJdbcNumPartitionsError(n: Int, jdbcNumPartitions: String): Throwable = {
+ new IllegalArgumentException(
+ s"Invalid value `$n` for parameter `$jdbcNumPartitions` in table writing " +
+ "via JDBC. The minimum value is 1.")
+ }
+
+ def transactionUnsupportedByJdbcServerError(): Throwable = {
+ new SQLFeatureNotSupportedException("The target JDBC server does not support " +
+ "transaction and can only support ALTER TABLE with a single action.")
+ }
+
+ def dataTypeUnsupportedYetError(dataType: DataType): Throwable = {
+ new UnsupportedOperationException(s"$dataType is not supported yet.")
+ }
+
+ def unsupportedOperationForDataTypeError(dataType: DataType): Throwable = {
+ new UnsupportedOperationException(s"DataType: ${dataType.catalogString}")
+ }
+
+ def inputFilterNotFullyConvertibleError(own: String): Throwable = {
+ new SparkException(s"The input filter of $own should be fully convertible.")
+ }
+
+ def cannotReadFooterForFileError(file: Path, e: IOException): Throwable = {
+ new SparkException(s"Could not read footer for file: $file", e)
+ }
+
+ def cannotReadFooterForFileError(file: FileStatus, e: RuntimeException): Throwable = {
+ new Exception(s"Could not read footer for file: $file", e)
Review comment:
Yes. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31757: [SPARK-33602][SQL] Group exception messages in execution/datasources
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31757:
URL: https://github.com/apache/spark/pull/31757#issuecomment-791358955
**[Test build #135810 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135810/testReport)** for PR 31757 at commit [`47bc26b`](https://github.com/apache/spark/commit/47bc26bcec5173a1ca5493492848eac7e71374f9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org