[GitHub] spark pull request #23215: [SPARK-26263][SQL] Validate partition values with...

[viirya <gi...@git.apache.org>]

Github user viirya commented on a diff in the pull request:

    https://github.com/apache/spark/pull/23215#discussion_r239388954
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala ---
    @@ -272,9 +279,13 @@ object PartitioningUtils {
           val literal = if (userSpecifiedDataTypes.contains(columnName)) {
             // SPARK-26188: if user provides corresponding column schema, get the column value without
             //              inference, and then cast it as user specified data type.
    -        val columnValue = inferPartitionColumnValue(rawColumnValue, false, timeZone)
    -        val castedValue =
    -          Cast(columnValue, userSpecifiedDataTypes(columnName), Option(timeZone.getID)).eval()
    +        val dataType = userSpecifiedDataTypes(columnName)
    +        val columnValueLiteral = inferPartitionColumnValue(rawColumnValue, false, timeZone)
    +        val columnValue = columnValueLiteral.eval()
    +        val castedValue = Cast(columnValueLiteral, dataType, Option(timeZone.getID)).eval()
    +        if (validatePartitionColumns && columnValue != null && castedValue == null) {
    +          throw new RuntimeException(s"Failed to cast partition value `$columnValue` to $dataType")
    --- End diff --
    
    Can we also show `columnName` in this exception message? So it is easier to know which partition column has such error.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org