[jira] [Created] (SPARK-60027) Partition transforms skip the input casts their bound function's inputTypes() asks for

["Peter Toth (Jira)" <ji...@apache.org>]

Peter Toth created SPARK-60027:
----------------------------------

             Summary: Partition transforms skip the input casts their bound function's inputTypes() asks for
                 Key: SPARK-60027
                 URL: https://issues.apache.org/jira/browse/SPARK-60027
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.2.0, 4.1.3, 4.0.4
            Reporter: Peter Toth


{{TransformExpression.resolvedFunction}} builds the bound function's call over the transform's raw children ({{V2ExpressionUtils.resolveScalarFunction}}). Nothing casts them to the function's {{inputTypes()}}, although the {{BoundFunction}} Javadoc says Spark will cast input values to the required data types. Only the write path coerces them ({{TypeCoercionExecutor}} in {{DistributionAndOrderingUtils}}).

The keyed shuffle evaluates this call to place the other side's rows (the {{KeyedPartitioning}} case of {{ShuffleExchangeExec}}). So a function whose input types differ from the column's type fails there.

Example with the test catalog's {{bucket}}, whose {{inputTypes()}} is {{(int, bigint)}}. Measured on master:
* table {{t}} with an {{int}} column {{id}}, partitioned by {{bucket(4, id)}};
* table {{u}} with an {{int}} column {{id}}, not partitioned;
* {{spark.sql.sources.v2.bucketing.shuffle.enabled=true}};
* {{SELECT /*+ MERGE(t, u) */ t.id, u.data FROM t JOIN u ON t.id = u.id}}.

The task fails:
{code}
java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long (java.lang.Integer and java.lang.Long are in module java.base of loader 'bootstrap')
	at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$.$anonfun$prepareShuffleDependency$14(ShuffleExchangeExec.scala:487)
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org