[jira] [Created] (SPARK-59114) java.lang.ClassCastException: class org.apache.spark.sql.catalyst.expressions.UnsafeRow cannot be cast to class org.apache.spark.sql.vectorized.ColumnarBatch

["Asif (Jira)" <ji...@apache.org>]

Asif created SPARK-59114:
----------------------------

             Summary: java.lang.ClassCastException: class org.apache.spark.sql.catalyst.expressions.UnsafeRow cannot be cast to class org.apache.spark.sql.vectorized.ColumnarBatch 
                 Key: SPARK-59114
                 URL: https://issues.apache.org/jira/browse/SPARK-59114
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.1.3, 4.2.0, 4.3.0
            Reporter: Asif


When a spark plan  using a FileSourceScanExec as leaf, reading a parquet format file, with default config (i.e parquet vector read enabled) is cached,  the SparkPlan created is something like

WholeStageCodegenExec -> ColumnToRowExec -> InputAdapter -> FileSourceScanExec.

The presence of ColumntoRowExec, implies that the child is going to output ColumnarBatch as the RDD.

This SparkPlan is stored in the CacheManager, and since the CachedRDDBuilder's buffers are lazy init, the spark plan is not executed at the time of caching.

 

Now after cached entry is created, and a query is executed involving the cached plan, with SQLConf having 
SQLConf.PARQUET_VECTORIZED_READER_ENABLED -> false,
 
then the cached plan is executed, with the this property, and as a result, the FileSourceScanExec no longer returns RDD[ColumnarBatch] ( it is the RDD of Row now).
While the CachedRDDBuilder code is expecting ColumnarBatch.
 
The fix proposed though is trivial, that is in the ColumnToRowExec.inputRDDs , explictly set the conf property of SQLConf.PARQUET_VECTORIZED_READER_ENABLED  as true.
The reasoning is that , since the plan contains ColumnToRowExec, that itself implies that the child RDD should be returning a ColumnarBatch and not Row.
However , I am not sure if this is sufficient and may be ColumnToRowExec.execute etc may also need to be tackled.
 
It is also not an elegant solution , more of work around.
The holistic solution should I suppose audit all the SQLConf properties which otherwise can impact the Plan being used , such that it renders it totally incompatible, May be such properties should be intialized in the Plan's constructir itself at the time of creation and it should over-ride any existing SQLConf.
 
I am exploring if its possible to store the nature of RDD return type ( Row or Batch), as part of FileSourceScanExec constructor itself, so that its unaffected by the subsequent change in conf.
 
A PR ( or may be two possible approaches fix will be filed shortly).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org