[GitHub] [hudi] melin edited a comment on pull request #3580: [HUDI-1869] Upgrading Spark3 To 3.1

[GitBox <gi...@apache.org>]

melin edited a comment on pull request #3580:
URL: https://github.com/apache/hudi/pull/3580#issuecomment-911319873


   spark 3.1.2
   ```
   spark-sql> select * from test_hudi_table;
   162943 [Executor task launch worker for task 0.0 in stage 32.0 (TID 1812)] ERROR org.apache.spark.executor.Executor  - Exception in task 0.0 in stage 32.0 (TID 1812)
   java.lang.ClassCastException: scala.Some cannot be cast to org.apache.spark.sql.catalyst.InternalRow
           at org.apache.spark.sql.hudi.HoodieMergeOnReadRDD$$anon$3.hasNext(HoodieMergeOnReadRDD.scala:240)
           at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
           at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
           at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755)
           at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:345)
           at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
           at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
           at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
           at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
           at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
           at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
           at org.apache.spark.scheduler.Task.run(Task.scala:131)
           at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498)
           at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
           at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
   ```
   Do you test use this PR? I think this PR has solved this exception by introduce the `HooodieAvroDeserializer` to match the `Option` result return by spark 3.1. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org