You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/11/10 06:29:36 UTC
[GitHub] [hudi] shenbinglife opened a new issue #2239: [SUPPORT] NoClassDefFoundError: org/apache/hudi/org/apache/commons/codec/binary/Base64
shenbinglife opened a new issue #2239:
URL: https://github.com/apache/hudi/issues/2239
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
Spark driver throw NoClassDefFoundError: org/apache/hudi/org/apache/commons/codec/binary/Base64
**To Reproduce**
Steps to reproduce the behavior:
1. add hudi-spark-bundle_2.11-0.5.3.jar to all spark node's path: /opt/spark/jars
2.create a simple main method , it contains "new BasicScheme()"
3.spark-submit --deploy-mode cluster xxx.jar
4. boom!!!
`20/11/10 13:56:10 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NoClassDefFoundError: org/apache/hudi/org/apache/commons/codec/binary/Base64
java.lang.NoClassDefFoundError: org/apache/hudi/org/apache/commons/codec/binary/Base64
at org.apache.http.impl.auth.BasicScheme.<init>(BasicScheme.java:65)
at org.apache.http.impl.auth.BasicScheme.<init>(BasicScheme.java:84)
at com.example.HttpConflict.main(HttpConflict.java:20)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.org.apache.commons.codec.binary.Base64
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
`
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.5.3
* Spark version : 2.4.0
* Hive version :
* Hadoop version : 3.0.0
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2239: [SUPPORT] NoClassDefFoundError: org/apache/hudi/org/apache/commons/codec/binary/Base64
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2239:
URL: https://github.com/apache/hudi/issues/2239#issuecomment-744703315
Closing this in favor of PR which was opened.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar closed issue #2239: [SUPPORT] NoClassDefFoundError: org/apache/hudi/org/apache/commons/codec/binary/Base64
Posted by GitBox <gi...@apache.org>.
bvaradar closed issue #2239:
URL: https://github.com/apache/hudi/issues/2239
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] sbernauer commented on issue #2239: [SUPPORT] NoClassDefFoundError: org/apache/hudi/org/apache/commons/codec/binary/Base64
Posted by GitBox <gi...@apache.org>.
sbernauer commented on issue #2239:
URL: https://github.com/apache/hudi/issues/2239#issuecomment-741761377
Hi @bvaradar,
we run into the same issue. Using hudi from master branch from 16th September (so version > 0.6.0). We run into the issue when syncing to a hive server using thrift over http enabling the following feature (thrift via TCP is no problem):
`hoodie.datasource.hive_sync.jdbcurl=jdbc:hive2://my-service.my-namespace.svc.cluster.local:10001/default;transportMode=http;ssl=false;httpPath=cliservice`
The jar /opt/spark/jars/commons-codec-1.10.jar is on the classpath. Do you have an idea why it is looking for org.apache.hudi.org.apache.commons.codec.binary.Base64 and not org.apache.commons.codec.binary.Base64?
The stacktrace is as following:
```
Exception in thread "main" org.apache.hudi.exception.HoodieException: java.lang.NoClassDefFoundError: org/apache/hudi/org/apache/commons/codec/binary/Base64
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$1(HoodieDeltaStreamer.java:152)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:147)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:464)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/apache/hudi/org/apache/commons/codec/binary/Base64
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.apache.hudi.async.AbstractAsyncService.waitForShutdown(AbstractAsyncService.java:79)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$1(HoodieDeltaStreamer.java:150)
... 15 more
Caused by: java.lang.NoClassDefFoundError: org/apache/hudi/org/apache/commons/codec/binary/Base64
at org.apache.http.impl.auth.BasicScheme.authenticate(BasicScheme.java:168)
at org.apache.hive.jdbc.HttpBasicAuthInterceptor.addHttpAuthHeader(HttpBasicAuthInterceptor.java:53)
at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:77)
at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:182)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:117)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
at org.apache.hive.service.rpc.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:170)
at org.apache.hive.service.rpc.thrift.TCLIService$Client.OpenSession(TCLIService.java:162)
at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:728)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:232)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.hudi.hive.HoodieHiveClient.createHiveConnection(HoodieHiveClient.java:419)
at org.apache.hudi.hive.HoodieHiveClient.<init>(HoodieHiveClient.java:95)
at org.apache.hudi.hive.HiveSyncTool.<init>(HiveSyncTool.java:66)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncMeta(DeltaSync.java:506)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:430)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:249)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:579)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.org.apache.commons.codec.binary.Base64
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 33 more
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on issue #2239: [SUPPORT] NoClassDefFoundError: org/apache/hudi/org/apache/commons/codec/binary/Base64
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #2239:
URL: https://github.com/apache/hudi/issues/2239#issuecomment-743028556
>org.apache.hudi.org.apache.commons.codec.binary.Base64 and not org.apache.commons.codec.binary.Base64?
@sbernauer this is because it's getting shaded in the bundles I suppose. Let me review your PR more closely. We try to rely as much as possible on the underlying engine's jars.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2239: [SUPPORT] NoClassDefFoundError: org/apache/hudi/org/apache/commons/codec/binary/Base64
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2239:
URL: https://github.com/apache/hudi/issues/2239#issuecomment-724875552
@shenbinglife : Are you seeing the same issue with 0.6.0 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org