You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Cosmin Iordache (Jira)" <ji...@apache.org> on 2022/12/02 08:37:00 UTC
[jira] [Commented] (HUDI-3744) NoSuchMethodError of getReadStatistics with Spark 3.2/Hadoop 3.2 using HBase
[ https://issues.apache.org/jira/browse/HUDI-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17642357#comment-17642357 ]
Cosmin Iordache commented on HUDI-3744:
---------------------------------------
I have the same error:
*Description of the issue follows:* * I'm using {{hudi-spark3-bundle-2.12}} on {{spark=3.3.1}} with {{hadoop=3.3.4}} .
* The error happens on a COW table when we do a couple of ingestions - apparently the {{view}} is not synchronized with the client . So at this place : {{RequestHandler.syncIfLocalViewBehind(RequestHandler.java:163)}}
*What happens:*
The reason is that {{FSDataInputStreamWrapper}} that is shaded in hudi comes from {{hbase=2.4.9}} this hbase that is used for hudi compilation is itself compiled against {{hadoop=2.10.1}} which has the following definition for the method {{getReadStatistics()}}
[https://github.com/apache/hadoop/blob/branch-2.10/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsDataInputStream.java#L106]
While in {{hadoop=3.3.4}} the definition is :
[https://github.com/apache/hadoop/blob/branch-3.3/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsDataInputStream.java#L107]
The code in the {{FSDataInputStreamWrapper}} directly references the {{hadoop=2.10}} version of the {{getReadStatistics()}} :
This is a reference to {{DFSInputStream$ReadStatistics}} in the decompiled {{FSDataInputStreamWrapper}} that comes with {{hudi-spark3-bundle-2.12}}
#203 = Utf8 getReadStatistics
#204 = Utf8 ()Lorg/apache/hadoop/hdfs/DFSInputStream$ReadStatistics;
#205 = NameAndType #203:#204 // getReadStatistics:()Lorg/apache/hadoop/hdfs/DFSInputStream$ReadStatistics;
#206 = Methodref #197.#205 // org/apache/hadoop/hdfs/client/HdfsDataInputStream.getReadStatistics:()Lorg/apache/hadoop/hdfs/DFSInputStream$ReadStatistics;
Stacktrace:
{code:java}
2022-12-01/14:57:52.770 [qtp700018681-1038] INFO o.a.h.c.t.t.HoodieActiveTimeline - Loaded instants upto : Option{val=[20221201145745641__deltacommit__COMPLETED]}2022-12-01/14:57:52.770 [qtp700018681-1038] DEBUG o.a.h.t.service.RequestHandler - TimeTakenMillis[Total=7, Refresh=0, handle=0, Check=0], Success=true, Query=basepath=hdfs%3A%2F%2Fnamenode%3A8020%2Fdata%2Flake%2F20221201-145418-342%2F1669902858975_112%2Fconverted&lastinstantts=20221201145745641&timelinehash=9a4be98105ed813c7483792025cface70d7d7d51ddeef6f7565d119eb176da9a, Host=192.168.15.158:38537, synced=false2022-12-01/14:57:52.770 [qtp700018681-1038] ERROR io.javalin.Javalin - Exception occurred while servicing http-requestjava.lang.NoSuchMethodError: 'org.apache.hadoop.hdfs.DFSInputStream$ReadStatistics org.apache.hadoop.hdfs.client.HdfsDataInputStream.getReadStatistics()' at org.apache.hudi.org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.updateInputStreamStatistics(FSDataInputStreamWrapper.java:249) at org.apache.hudi.org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.close(FSDataInputStreamWrapper.java:296) at org.apache.hudi.org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.closeStreams(HFileBlock.java:1825) at org.apache.hudi.org.apache.hadoop.hbase.io.hfile.HFilePreadReader.close(HFilePreadReader.java:107) at org.apache.hudi.org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.close(HFileReaderImpl.java:1421) at org.apache.hudi.io.storage.HoodieHFileReader.close(HoodieHFileReader.java:218) at org.apache.hudi.metadata.HoodieBackedTableMetadata.closeReader(HoodieBackedTableMetadata.java:589) at org.apache.hudi.metadata.HoodieBackedTableMetadata.close(HoodieBackedTableMetadata.java:572) at org.apache.hudi.metadata.HoodieBackedTableMetadata.closePartitionReaders(HoodieBackedTableMetadata.java:580) at org.apache.hudi.metadata.HoodieBackedTableMetadata.reset(HoodieBackedTableMetadata.java:647) at org.apache.hudi.metadata.HoodieMetadataFileSystemView.sync(HoodieMetadataFileSystemView.java:77) at org.apache.hudi.timeline.service.RequestHandler.syncIfLocalViewBehind(RequestHandler.java:163) at org.apache.hudi.timeline.service.RequestHandler.access$100(RequestHandler.java:68) at org.apache.hudi.timeline.service.RequestHandler$ViewHandler.handle(RequestHandler.java:494) at io.javalin.security.SecurityUtil.noopAccessManager(SecurityUtil.kt:22) at io.javalin.Javalin.lambda$addHandler$0(Javalin.java:606) at io.javalin.core.JavalinServlet$service$2$1.invoke(JavalinServlet.kt:46) at io.javalin.core.JavalinServlet$service$2$1.invoke(JavalinServlet.kt:17) at io.javalin.core.JavalinServlet$service$1.invoke(JavalinServlet.kt:143) at io.javalin.core.JavalinServlet$service$2.invoke(JavalinServlet.kt:41) at io.javalin.core.JavalinServlet.service(JavalinServlet.kt:107) at io.javalin.core.util.JettyServerUtil$initialize$httpHandler$1.doHandle(JettyServerUtil.kt:72) at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.apache.hudi.org.apache.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.apache.hudi.org.apache.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668) at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.apache.hudi.org.apache.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.apache.hudi.org.apache.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.apache.hudi.org.apache.jetty.server.handler.HandlerList.handle(HandlerList.java:61) at org.apache.hudi.org.apache.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174) at org.apache.hudi.org.apache.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.apache.hudi.org.apache.jetty.server.Server.handle(Server.java:502) at org.apache.hudi.org.apache.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.apache.hudi.org.apache.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.apache.hudi.org.apache.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.apache.hudi.org.apache.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.apache.hudi.org.apache.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.apache.hudi.org.apache.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at org.apache.hudi.org.apache.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.apache.hudi.org.apache.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base/java.lang.Thread.run(Thread.java:829) {code}
> NoSuchMethodError of getReadStatistics with Spark 3.2/Hadoop 3.2 using HBase
> -----------------------------------------------------------------------------
>
> Key: HUDI-3744
> URL: https://issues.apache.org/jira/browse/HUDI-3744
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Ethan Guo
> Assignee: Raymond Xu
> Priority: Blocker
> Fix For: 0.11.0
>
>
> Environment: Hadoop 3.2.1 & Spark-3.2.1
> hudi compile from commit f2a93ead3b5a6964a72b3543ada58aa334edef9c
> just use spark-sql and default job configuration to execute "show partitions [hudi_table_name];"
> {code:java}
> // command
> spark-sql --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog
> // spark-sql
> spark-sql> show partitions hudi_partition_table;
> {code}
> // code placeholderjava.lang.NoSuchMethodError: org.apache.hadoop.hdfs.client.HdfsDataInputStream.getReadStatistics()Lorg/apache/hadoop/hdfs/DFSInputStream$ReadStatistics;
> at org.apache.hudi.org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.updateInputStreamStatistics(FSDataInputStreamWrapper.java:249)
> at org.apache.hudi.org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.close(FSDataInputStreamWrapper.java:296)
> at org.apache.hudi.org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.closeStreams(HFileBlock.java:1825)
> at org.apache.hudi.org.apache.hadoop.hbase.io.hfile.HFilePreadReader.close(HFilePreadReader.java:107)
> at org.apache.hudi.org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.close(HFileReaderImpl.java:1421)
> at org.apache.hudi.io.storage.HoodieHFileReader.close(HoodieHFileReader.java:423)
> at org.apache.hudi.metadata.HoodieBackedTableMetadata.close(HoodieBackedTableMetadata.java:435)
> at org.apache.hudi.metadata.HoodieBackedTableMetadata.lambda$getRecordsByKeys$0(HoodieBackedTableMetadata.java:162)
> at java.util.HashMap.forEach(HashMap.java:1290)
> at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordsByKeys(HoodieBackedTableMetadata.java:138)
> at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordByKey(HoodieBackedTableMetadata.java:128)
> at org.apache.hudi.metadata.BaseTableMetadata.fetchAllPartitionPaths(BaseTableMetadata.java:281)
> at org.apache.hudi.metadata.BaseTableMetadata.getAllPartitionPaths(BaseTableMetadata.java:111)
> at org.apache.hudi.common.fs.FSUtils.getAllPartitionPaths(FSUtils.java:308)
> at org.apache.spark.sql.hudi.HoodieSqlCommonUtils$.getAllPartitionPaths(HoodieSqlCommonUtils.scala:81)
> at org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable.getPartitionPaths(HoodieCatalogTable.scala:157)
> at org.apache.spark.sql.hudi.command.ShowHoodieTablePartitionsCommand.run(ShowHoodieTablePartitionsCommand.scala:51)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
> at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:110)
> at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
> at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
> at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
> at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
> at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:110)
> at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:106)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)