You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Sagar Sumit (Jira)" <ji...@apache.org> on 2022/04/12 03:26:00 UTC
[jira] [Created] (HUDI-3857) Long running deltastreamer test with async compaction fails on EMR spark
Sagar Sumit created HUDI-3857:
---------------------------------
Summary: Long running deltastreamer test with async compaction fails on EMR spark
Key: HUDI-3857
URL: https://issues.apache.org/jira/browse/HUDI-3857
Project: Apache Hudi
Issue Type: Test
Reporter: Sagar Sumit
Assignee: Sagar Sumit
Fix For: 0.11.0
EMR 6.5, Spark 3.1.2
While running continuous deltastreamer with async compaction enabled, I hit this exception
{code:java}
Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.PartitionedFile.<init>(Lorg/apache/spark/sql/catalyst/InternalRow;Ljava/lang/String;JJ[Ljava/lang/String;)V
at org.apache.hudi.MergeOnReadSnapshotRelation.$anonfun$buildSplits$2(MergeOnReadSnapshotRelation.scala:130)
at scala.Option.map(Option.scala:230)
at org.apache.hudi.MergeOnReadSnapshotRelation.$anonfun$buildSplits$1(MergeOnReadSnapshotRelation.scala:128)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike.map(TraversableLike.scala:238)
at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
at scala.collection.immutable.List.map(List.scala:298)
at org.apache.hudi.MergeOnReadSnapshotRelation.buildSplits(MergeOnReadSnapshotRelation.scala:124)
at org.apache.hudi.MergeOnReadSnapshotRelation.collectFileSplits(MergeOnReadSnapshotRelation.scala:108)
at org.apache.hudi.MergeOnReadSnapshotRelation.collectFileSplits(MergeOnReadSnapshotRelation.scala:44)
at org.apache.hudi.HoodieBaseRelation.buildScan(HoodieBaseRelation.scala:221) {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)