You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Alexey Kudinkin (Jira)" <ji...@apache.org> on 2022/02/01 22:54:00 UTC
[jira] [Commented] (HUDI-2899) Fix DataFormatter usages removed in Spark 3.2
[ https://issues.apache.org/jira/browse/HUDI-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17485520#comment-17485520 ]
Alexey Kudinkin commented on HUDI-2899:
---------------------------------------
Yep, it's working fine w/ Spark 3.2 now
> Fix DataFormatter usages removed in Spark 3.2
> ---------------------------------------------
>
> Key: HUDI-2899
> URL: https://issues.apache.org/jira/browse/HUDI-2899
> Project: Apache Hudi
> Issue Type: Bug
> Components: spark
> Reporter: Alexey Kudinkin
> Assignee: Yann Byron
> Priority: Major
> Fix For: 0.11.0
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
>
> Trying to read that is partitioned on a string field ("product_category") rather than by date, leads to `NoSuchMethodError`
> {code:java}
> scala> val readDf: DataFrame =
> | spark.read.option(DataSourceReadOptions.ENABLE_DATA_SKIPPING.key(), "false").format("hudi").load(outputPath)
> java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.util.DateFormatter$.apply(Ljava/time/ZoneId;)Lorg/apache/spark/sql/catalyst/util/DateFormatter;
> at org.apache.spark.sql.execution.datasources.Spark3ParsePartitionUtil.parsePartition(Spark3ParsePartitionUtil.scala:32)
> at org.apache.hudi.HoodieFileIndex.$anonfun$getAllQueryPartitionPaths$3(HoodieFileIndex.scala:559)
> at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> at scala.collection.TraversableLike.map(TraversableLike.scala:286)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
> at scala.collection.AbstractTraversable.map(Traversable.scala:108)
> at org.apache.hudi.HoodieFileIndex.getAllQueryPartitionPaths(HoodieFileIndex.scala:511)
> at org.apache.hudi.HoodieFileIndex.loadPartitionPathFiles(HoodieFileIndex.scala:575)
> at org.apache.hudi.HoodieFileIndex.refresh0(HoodieFileIndex.scala:360)
> at org.apache.hudi.HoodieFileIndex.<init>(HoodieFileIndex.scala:157)
> at org.apache.hudi.DefaultSource.getBaseFileOnlyView(DefaultSource.scala:199)
> at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:119)
> at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:69)
> at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:350)
> at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:274)
> at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:245)
> at scala.Option.getOrElse(Option.scala:189)
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:245)
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:188)
> ... 68 elided {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)