You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/15 12:23:15 UTC
[GitHub] [hudi] AndrOidN13 opened a new issue #4001: [SUPPORT] Hudi 0.9.0 fails when used with Spark 3.2.0
AndrOidN13 opened a new issue #4001:
URL: https://github.com/apache/hudi/issues/4001
**Describe the problem you faced**
When used with Spark 3.2.0 Hudi 0.9.0 fails when attempting to read Hudi data with `spark.read.format("hudi").load(<path>)`
**To Reproduce**
Steps to reproduce the behavior:
1. Try reading Hudi data with Spark 3.2.0 and Hudi 0.9.0
**Expected behavior**
Hudi data is loaded.
**Environment Description**
* Hudi version : 0.9.0
* Spark version : 3.2.0
* Hive version : 2.3.9
* Hadoop version : 3.3.1
* Storage (HDFS/S3/GCS..) : local fs( macOS Catalina)
* Running on Docker? (yes/no) : no
**Stacktrace**
```
java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.util.DateFormatter$.apply(Ljava/time/ZoneId;)Lorg/apache/spark/sql/catalyst/util/DateFormatter;
at org.apache.spark.sql.execution.datasources.Spark3ParsePartitionUtil.parsePartition(Spark3ParsePartitionUtil.scala:32)
at org.apache.hudi.HoodieFileIndex.$anonfun$getAllQueryPartitionPaths$3(HoodieFileIndex.scala:404)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:273)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.map(TraversableLike.scala:273)
at scala.collection.TraversableLike.map$(TraversableLike.scala:266)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at org.apache.hudi.HoodieFileIndex.getAllQueryPartitionPaths(HoodieFileIndex.scala:356)
at org.apache.hudi.HoodieFileIndex.loadPartitionPathFiles(HoodieFileIndex.scala:420)
at org.apache.hudi.HoodieFileIndex.refresh0(HoodieFileIndex.scala:214)
at org.apache.hudi.HoodieFileIndex.<init>(HoodieFileIndex.scala:149)
at org.apache.hudi.DefaultSource.getBaseFileOnlyView(DefaultSource.scala:201)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:117)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:67)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:350)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:274)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:245)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:245)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:188)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4001: [SUPPORT] Hudi 0.9.0 fails when used with Spark 3.2.0
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4001:
URL: https://github.com/apache/hudi/issues/4001#issuecomment-974691214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed issue #4001: [SUPPORT] Hudi 0.9.0 fails when used with Spark 3.2.0
Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #4001:
URL: https://github.com/apache/hudi/issues/4001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4001: [SUPPORT] Hudi 0.9.0 fails when used with Spark 3.2.0
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4001:
URL: https://github.com/apache/hudi/issues/4001#issuecomment-974704505
tracking the problem in https://issues.apache.org/jira/browse/HUDI-2811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed issue #4001: [SUPPORT] Hudi 0.9.0 fails when used with Spark 3.2.0
Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #4001:
URL: https://github.com/apache/hudi/issues/4001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rubenssoto commented on issue #4001: [SUPPORT] Hudi 0.9.0 fails when used with Spark 3.2.0
Posted by GitBox <gi...@apache.org>.
rubenssoto commented on issue #4001:
URL: https://github.com/apache/hudi/issues/4001#issuecomment-971064195
I think Hudi doesn't has official support for Spark 3.2, master branch and the future version 0.10 has support for spark 3.1
https://github.com/apache/hudi/pull/3844
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4001: [SUPPORT] Hudi 0.9.0 fails when used with Spark 3.2.0
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4001:
URL: https://github.com/apache/hudi/issues/4001#issuecomment-974691214
yes Spark 3.2 not officially supported yet. But in the meantime, we'd like to collect info and hoping to get it working in 0.11.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org