You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "xi chaomin (Jira)" <ji...@apache.org> on 2022/09/16 09:15:00 UTC
[jira] [Comment Edited] (HUDI-3983) ClassNotFoundException when using hudi-spark-bundle to write table with hbase index
[ https://issues.apache.org/jira/browse/HUDI-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605706#comment-17605706 ]
xi chaomin edited comment on HUDI-3983 at 9/16/22 9:14 AM:
-----------------------------------------------------------
I tried the method mentioned in [https://github.com/apache/hudi/issues/6623]
According the aws doc [https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hudi-considerations.html], I submit the job with dependencies in hudi-cli/taget/lib, the write is successful.
In other words, the job succeed with original hbase dependencies. :(
was (Author: xichaomin):
I tried the method mentioned in [https://github.com/apache/hudi/issues/6623]
According the aws doc https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hudi-considerations.html, I submit the job with dependencies in hudi-cli/taget/lib, the write is
successful.
In other words, the job succeed with original hbase dependencies. :(
> ClassNotFoundException when using hudi-spark-bundle to write table with hbase index
> -----------------------------------------------------------------------------------
>
> Key: HUDI-3983
> URL: https://issues.apache.org/jira/browse/HUDI-3983
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: xi chaomin
> Assignee: Sagar Sumit
> Priority: Critical
> Fix For: 0.12.1
>
>
> I ran a spark job and encountered several ClassNotFoundExceptions. spark version is 3.1 and scala version is 2.12.
> 1.
> {code:java}
> java.lang.NoClassDefFoundError: org/apache/hudi/org/apache/hadoop/hbase/protobuf/generated/AuthenticationProtos$TokenIdentifier$Kind
> at org.apache.hudi.org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.translateException(RpcRetryingCallerImpl.java:222)
> at org.apache.hudi.org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:195)
> at org.apache.hudi.org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:395)
> at org.apache.hudi.org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369)
> at org.apache.hudi.org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:108) {code}
> including org.apache.hbase:hbase-protocol in packaging/hudi-spark-bundle/pom.xml can solve this error.
> 2.
> {code:java}
> java.lang.ClassNotFoundException: org.apache.hudi.org.apache.hbase.thirdparty.com.google.gson.GsonBuilder
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) {code}
> including org.apache.hbase.thirdparty:hbase-shaded-gson n packaging/hudi-spark-bundle/pom.xml can solve this error.
> 3.
> {code:java}
> java.lang.ClassNotFoundException: Class org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener not found {code}
> There is a configuration in hbase-site.xml
> {code:java}
> <property>
> <name>hbase.status.listener.class&amp;amp;amp;amp;lt;/name>
> <value>org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener</value>
> <description>
> Implementation of the status listener with a multicast message.
> </description>
> </property> {code}
> I set _*hbase.status.listener.class*_ to _*org.apache.hudi.org.apache.hadoop.hbase.client.ClusterStatusListener$MulticastListener*_ in hbase configureation, the ClassNotFoundException has resolved, but get another exception
> {code:java}
> org.apache.hudi.org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
> 2022-08-26T07:12:57.603Z, RpcRetryingCaller{globalStartTime=2022-08-26T07:12:56.651Z, pause=100, maxAttempts=1}, org.apache.hudi.org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Call to address=x.x.x.x/x.x.x.x:16020 failed on local exception: org.apache.hudi.org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection closed at org.apache.hudi.org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:146)
> at org.apache.hudi.org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hudi.org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Call to address=x.x.x.x/x.x.x.x:16020 failed on local exception: org.apache.hudi.org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection closed
> at org.apache.hudi.org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:214)
> at org.apache.hudi.org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:384)
> at org.apache.hudi.org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:89)
> at org.apache.hudi.org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:415)
> at org.apache.hudi.org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:411)
> at org.apache.hudi.org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:118)
> at org.apache.hudi.org.apache.hadoop.hbase.ipc.Call.setException(Call.java:133)
> at org.apache.hudi.org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.cleanupCalls(NettyRpcDuplexHandler.java:203)
> at org.apache.hudi.org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelInactive(NettyRpcDuplexHandler.java:211)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at org.apache.hudi.org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> ... 1 more
> Caused by: org.apache.hudi.org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection closed
> ... 26 more{code}
> I remove the relocations related to hbase in packaging/hudi-spark-bundle/pom.xml , the job succeed.
> I checked the debug logs, but have no idea of the reason why ConnectionClosedException occurs when we use relocation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)