You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/05/01 16:52:54 UTC
[GitHub] [hudi] ismailsimsek opened a new issue #2910: java.lang.NoSuchMethodError: 'java.lang.String org.apache.http.client.utils.URLEncodedUtils.formatSegments
ismailsimsek opened a new issue #2910:
URL: https://github.com/apache/hudi/issues/2910
Getting
`java.lang.RuntimeException: java.lang.NoSuchMethodError: 'java.lang.String org.apache.http.client.utils.URLEncodedUtils.formatSegments(java.lang.String[])'`
A clear and concise description of the problem.
dependency to `org.apache.httpcomponents:httpclient` conflicts with embedded class in `hudi-spark-bundle_2.12`
im not expert on this but i belive `hudi-spark-bundle_2.12` should use shading for `httpclient`
**Environment Description**
Quarkus application
* Hudi version : 0.7 and 0.8
* Spark version : 3.0.2
* Hive version :
* Hadoop version : 3.2.0
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : NO
Add any other context about the problem here.
**Stacktrace**
```
java.lang.RuntimeException: java.lang.NoSuchMethodError: 'java.lang.String org.apache.http.client.utils.URLEncodedUtils.formatSegments(java.lang.String[])'
at io.quarkus.test.junit.QuarkusTestExtension.throwBootFailureException(QuarkusTestExtension.java:674)
at io.quarkus.test.junit.QuarkusTestExtension.interceptBeforeAllMethod(QuarkusTestExtension.java:731)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptLifecycleMethod(TimeoutExtension.java:126)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptBeforeAllMethod(TimeoutExtension.java:68)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllMethods$9(ClassBasedTestDescriptor.java:384)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllMethods(ClassBasedTestDescriptor.java:382)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:196)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:78)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:136)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:71)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
Caused by: java.lang.NoSuchMethodError: 'java.lang.String org.apache.http.client.utils.URLEncodedUtils.formatSegments(java.lang.String[])'
at org.apache.maven.wagon.shared.http.EncodingUtil.encodeURLToString(EncodingUtil.java:112)
at org.apache.maven.wagon.shared.http.EncodingUtil.encodeURLToString(EncodingUtil.java:95)
at org.apache.maven.wagon.shared.http.AbstractHttpClientWagon.buildUrl(AbstractHttpClientWagon.java:731)
at org.apache.maven.wagon.shared.http.AbstractHttpClientWagon.buildUrl(AbstractHttpClientWagon.java:720)
at org.apache.maven.wagon.shared.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:1146)
at org.apache.maven.wagon.shared.http.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:1138)
at org.apache.maven.wagon.StreamWagon.getInputStream(StreamWagon.java:126)
at org.apache.maven.wagon.StreamWagon.getIfNewer(StreamWagon.java:88)
at org.apache.maven.wagon.StreamWagon.get(StreamWagon.java:61)
at org.eclipse.aether.transport.wagon.WagonTransporter$GetTaskRunner.run(WagonTransporter.java:567)
at org.eclipse.aether.transport.wagon.WagonTransporter.execute(WagonTransporter.java:435)
at org.eclipse.aether.transport.wagon.WagonTransporter.get(WagonTransporter.java:412)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask(BasicRepositoryConnector.java:457)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:364)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:75)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:644)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:262)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:499)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:401)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:229)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:207)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:240)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:171)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor(DefaultRepositorySystem.java:255)
at io.quarkus.bootstrap.resolver.maven.MavenArtifactResolver.resolveDescriptorInternal(MavenArtifactResolver.java:189)
at io.quarkus.bootstrap.resolver.maven.MavenArtifactResolver.newCollectManagedRequest(MavenArtifactResolver.java:320)
at io.quarkus.bootstrap.resolver.maven.MavenArtifactResolver.collectManagedDependencies(MavenArtifactResolver.java:305)
at io.quarkus.bootstrap.resolver.maven.DeploymentInjectingDependencyVisitor.collectDependencies(DeploymentInjectingDependencyVisitor.java:181)
at io.quarkus.bootstrap.resolver.maven.DeploymentInjectingDependencyVisitor.injectDeploymentDependencies(DeploymentInjectingDependencyVisitor.java:85)
at io.quarkus.bootstrap.resolver.BootstrapAppModelResolver.doResolveModel(BootstrapAppModelResolver.java:236)
at io.quarkus.bootstrap.resolver.BootstrapAppModelResolver.resolveManagedModel(BootstrapAppModelResolver.java:152)
at io.quarkus.bootstrap.BootstrapAppModelFactory.resolveAppModel(BootstrapAppModelFactory.java:306)
at io.quarkus.bootstrap.app.QuarkusBootstrap.bootstrap(QuarkusBootstrap.java:163)
at io.quarkus.test.junit.QuarkusTestExtension.doJavaStart(QuarkusTestExtension.java:300)
at io.quarkus.test.junit.QuarkusTestExtension.ensureStarted(QuarkusTestExtension.java:643)
at io.quarkus.test.junit.QuarkusTestExtension.beforeAll(QuarkusTestExtension.java:689)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$8(ClassBasedTestDescriptor.java:368)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:368)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:192)
... 33 more
Process finished with exit code 255
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar closed issue #2910: java.lang.NoSuchMethodError: 'java.lang.String org.apache.http.client.utils.URLEncodedUtils.formatSegments
Posted by GitBox <gi...@apache.org>.
vinothchandar closed issue #2910:
URL: https://github.com/apache/hudi/issues/2910
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2910: java.lang.NoSuchMethodError: 'java.lang.String org.apache.http.client.utils.URLEncodedUtils.formatSegments
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2910:
URL: https://github.com/apache/hudi/issues/2910#issuecomment-854414277
@ismailsimsek unit test packages should have a `test` scope and show not be available in the bundles. Can you explain what you mean by `coming from quarks unit test` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2910: java.lang.NoSuchMethodError: 'java.lang.String org.apache.http.client.utils.URLEncodedUtils.formatSegments
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2910:
URL: https://github.com/apache/hudi/issues/2910#issuecomment-842015617
@ismailsimsek Thanks for providing these details. Could you please also let us know if the `httpclient-4.5.9.jar` jar is something that was part of your spark distribution or is this jar something was introduced as part of your application environment ?
@nsivabalan Looks like shading the http-client shouldn't hurt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ismailsimsek commented on issue #2910: java.lang.NoSuchMethodError: 'java.lang.String org.apache.http.client.utils.URLEncodedUtils.formatSegments
Posted by GitBox <gi...@apache.org>.
ismailsimsek commented on issue #2910:
URL: https://github.com/apache/hudi/issues/2910#issuecomment-832581095
the [project is here ](https://github.com/memiiso/debezium-server-batch)
it uses debezium and replicates database events to Hudi table as mini batches
the [pull request and pipeline error is here ](https://github.com/memiiso/debezium-server-batch/pull/11)
i believe this two dependencies are having conflict, i think hudi bundle loads the class but its bundled older version of httpclient library and missing the method
```
# ls -l
total 75504
-rw-rw-rw-@ 1 myuser staff 774384 May 5 12:17 httpclient-4.5.9.jar
-rw-rw-rw-@ 1 myuser staff 37820462 May 5 12:17 hudi-spark-bundle_2.12-0.8.0.jar
# for i in *.jar; do jar -tvf "$i" | grep -Hsi URLEncodedUtils && echo "$i"; done
(standard input): 16562 Fri Jun 07 15:24:26 CEST 2019 org/apache/http/client/utils/URLEncodedUtils.class
httpclient-4.5.9.jar
(standard input): 13664 Tue Apr 06 09:09:52 CEST 2021 org/apache/http/client/utils/URLEncodedUtils.class
hudi-spark-bundle_2.12-0.8.0.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2910: java.lang.NoSuchMethodError: 'java.lang.String org.apache.http.client.utils.URLEncodedUtils.formatSegments
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2910:
URL: https://github.com/apache/hudi/issues/2910#issuecomment-859329258
@ismailsimsek Is this still an issue ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2910: java.lang.NoSuchMethodError: 'java.lang.String org.apache.http.client.utils.URLEncodedUtils.formatSegments
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2910:
URL: https://github.com/apache/hudi/issues/2910#issuecomment-842015617
@ismailsimsek Thanks for providing these details. Could you please also let us know if the `httpclient-4.5.9.jar` jar is something that was part of your spark distribution or is this jar something was introduced as part of your application environment ?
@nsivabalan Looks like shading the http-client shouldn't hurt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #2910: java.lang.NoSuchMethodError: 'java.lang.String org.apache.http.client.utils.URLEncodedUtils.formatSegments
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #2910:
URL: https://github.com/apache/hudi/issues/2910#issuecomment-832054474
@ismailsimsek : is it possible to give us steps to reproduce this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2910: java.lang.NoSuchMethodError: 'java.lang.String org.apache.http.client.utils.URLEncodedUtils.formatSegments
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2910:
URL: https://github.com/apache/hudi/issues/2910#issuecomment-854414277
@ismailsimsek unit test packages should have a `test` scope and show not be available in the bundles. Can you explain what you mean by `coming from quarks unit test` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ismailsimsek commented on issue #2910: java.lang.NoSuchMethodError: 'java.lang.String org.apache.http.client.utils.URLEncodedUtils.formatSegments
Posted by GitBox <gi...@apache.org>.
ismailsimsek commented on issue #2910:
URL: https://github.com/apache/hudi/issues/2910#issuecomment-846409600
one dependency is coming from `org.apache.hadoop:hadoop-common:jar:3.2.0:compile`
i believe conflicting call/dependency is coming from quarks unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org