You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by Fabian Peters <po...@mercadu.de> on 2022/07/14 11:03:12 UTC
Rest transform on Beam: NoClassDefFoundError: javax/ws/rs/core/Link$Builder
Hi all!
I've started playing with Hop recently and do like quite a few things about it. Thanks for all the effort!
Unfortunately, there's one thing that's holding me back now: A pipeline containing the rest transform works fine when launched from the GUI using local or Beam-Direct. But it fails (see stack trace below) when launched via ./hop-run.sh using Beam-Direct and it always fails for DataFlow – which is what I want to use in the end.
For Beam-Direct, I've verified the failure on both my macOS machine and on a debian bullseye VM, with these java versions:
openjdk version "11.0.15" 2022-04-19
OpenJDK Runtime Environment Homebrew (build 11.0.15+0)
OpenJDK 64-Bit Server VM Homebrew (build 11.0.15+0, mixed mode)
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mode)
After some digging, my guess is that the rest transform implementation is based on jersey 1.19.4 but beam comes with jersey 2.3. I don't understand all the classloading magic going on, so for now I'm giving up.
I'll create a ticket on Jira, but I still wanted to hear whether there's something like an easy way out. Ideas very much welcome!
cheers
Fabian
2022/07/13 13:39:16 - General - Executing this pipeline using the Beam Pipeline Engine with run configuration 'Beam-Direct'
2022/07/13 13:39:18 - GET PLT alerts.0 - ERROR: Error initializing transform [GET PLT alerts]
2022/07/13 13:39:18 - GET PLT alerts.0 - ERROR: java.lang.NoClassDefFoundError: javax/ws/rs/core/Link$Builder
2022/07/13 13:39:18 - GET PLT alerts.0 - at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
2022/07/13 13:39:18 - GET PLT alerts.0 - at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
2022/07/13 13:39:18 - GET PLT alerts.0 - at java.base/java.lang.Class.getConstructor0(Class.java:3342)
2022/07/13 13:39:18 - GET PLT alerts.0 - at java.base/java.lang.Class.newInstance(Class.java:556)
2022/07/13 13:39:18 - GET PLT alerts.0 - at javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:65)
2022/07/13 13:39:18 - GET PLT alerts.0 - at javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:117)
2022/07/13 13:39:18 - GET PLT alerts.0 - at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:105)
2022/07/13 13:39:18 - GET PLT alerts.0 - at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91)
2022/07/13 13:39:18 - GET PLT alerts.0 - at javax.ws.rs.core.MediaType.<clinit>(MediaType.java:44)
2022/07/13 13:39:18 - GET PLT alerts.0 - at org.apache.hop.pipeline.transforms.rest.Rest.init(Rest.java:571)
2022/07/13 13:39:18 - GET PLT alerts.0 - at org.apache.hop.pipeline.transform.TransformInitThread.run(TransformInitThread.java:66)
2022/07/13 13:39:18 - GET PLT alerts.0 - at java.base/java.lang.Thread.run(Thread.java:829)
2022/07/13 13:39:18 - GET PLT alerts.0 - Caused by: java.lang.ClassNotFoundException: Unable to load class 'javax.ws.rs.core.Link$Builder' in this classloader or in the parent
2022/07/13 13:39:18 - GET PLT alerts.0 - at org.apache.hop.core.plugins.HopURLClassLoader.loadClass(HopURLClassLoader.java:96)
2022/07/13 13:39:18 - GET PLT alerts.0 - at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
2022/07/13 13:39:18 - GET PLT alerts.0 - ... 12 more
2022/07/13 13:39:18 - GET PLT alerts.0 - Caused by: java.lang.ClassNotFoundException: javax.ws.rs.core.Link$Builder
2022/07/13 13:39:18 - GET PLT alerts.0 - at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
2022/07/13 13:39:18 - GET PLT alerts.0 - at org.apache.hop.core.plugins.HopURLClassLoader.loadClassFromThisLoader(HopURLClassLoader.java:66)
2022/07/13 13:39:18 - GET PLT alerts.0 - at org.apache.hop.core.plugins.HopURLClassLoader.loadClass(HopURLClassLoader.java:91)
2022/07/13 13:39:18 - GET PLT alerts.0 - ... 13 more
2022/07/13 13:39:18 - ERROR: Transform [GET PLT alerts.0] failed to initialize!
2022/07/13 13:39:18 - General - ERROR: Error starting the Beam pipeline
2022/07/13 13:39:18 - General - ERROR: org.apache.hop.core.exception.HopException:
2022/07/13 13:39:18 - General - Error executing pipeline with runner Direct
2022/07/13 13:39:18 - General - java.lang.RuntimeException: Error executing TransformFn
2022/07/13 13:39:18 - General -
2022/07/13 13:39:18 - General - at org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:254)
2022/07/13 13:39:18 - General - at org.apache.hop.beam.engines.BeamPipelineEngine.lambda$startThreads$0(BeamPipelineEngine.java:301)
2022/07/13 13:39:18 - General - at java.base/java.lang.Thread.run(Thread.java:829)
2022/07/13 13:39:18 - General - Caused by: org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.RuntimeException: Error executing TransformFn
2022/07/13 13:39:18 - General - at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:373)
2022/07/13 13:39:18 - General - at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:341)
2022/07/13 13:39:18 - General - at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:218)
2022/07/13 13:39:18 - General - at org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:242)
2022/07/13 13:39:18 - General - ... 2 more
2022/07/13 13:39:18 - General - Caused by: java.lang.RuntimeException: Error executing TransformFn
2022/07/13 13:39:18 - General - at org.apache.hop.beam.core.transform.TransformTransform$TransformFn.processElement(TransformTransform.java:571)
2022/07/13 13:39:18 - General - Caused by: org.apache.hop.core.exception.HopException:
2022/07/13 13:39:18 - General - We failed to initialize at least one transform. Execution can not begin!
2022/07/13 13:39:18 - General -
2022/07/13 13:39:18 - General -
2022/07/13 13:39:18 - General - at org.apache.hop.pipeline.Pipeline.prepareExecution(Pipeline.java:1026)
2022/07/13 13:39:18 - General - at org.apache.hop.pipeline.engines.local.LocalPipelineEngine.prepareExecution(LocalPipelineEngine.java:206)
2022/07/13 13:39:18 - General - at org.apache.hop.beam.core.transform.TransformTransform$TransformFn.processElement(TransformTransform.java:441)
2022/07/13 13:39:18 - General - at org.apache.hop.beam.core.transform.TransformTransform$TransformFn$DoFnInvoker.invokeProcessElement(Unknown Source)
2022/07/13 13:39:18 - General - at org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:228)
2022/07/13 13:39:18 - General - at org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:184)
2022/07/13 13:39:18 - General - at org.apache.beam.repackaged.direct_java.runners.core.SimplePushbackSideInputDoFnRunner.processElementInReadyWindows(SimplePushbackSideInputDoFnRunner.java:79)
2022/07/13 13:39:18 - General - at org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:244)
2022/07/13 13:39:18 - General - at org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:54)
2022/07/13 13:39:18 - General - at org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:165)
2022/07/13 13:39:18 - General - at org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:129)
2022/07/13 13:39:18 - General - at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2022/07/13 13:39:18 - General - at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
2022/07/13 13:39:18 - General - at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2022/07/13 13:39:18 - General - at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2022/07/13 13:39:18 - General - at java.base/java.lang.Thread.run(Thread.java:829)
Re: Rest transform on Beam: NoClassDefFoundError: javax/ws/rs/core/Link$Builder
Posted by Hans Van Akelyen <ha...@gmail.com>.
Hi Fabian,
Thanks for testing things out, you have indeed found a situation where the
base Beam libraries are conflicting with our transforms.
When using the Hop engine we are able to keep the classpaths separate and
each transform/plugin is isolated in its own classloader.
When running in Beam we make a combination of all our jars and they end up
in the same classpath.
We have already put quite some time in aligning libraries for core
transforms so they work seamlessly in Beam.
I'll work on upgrading the jersey libraries so these transforms can work in
Beam too [1].
Cheers,
Hans
[1] https://issues.apache.org/jira/browse/HOP-4059
On Thu, 14 Jul 2022 at 13:03, Fabian Peters <po...@mercadu.de> wrote:
> Hi all!
>
> I've started playing with Hop recently and do like quite a few things
> about it. Thanks for all the effort!
>
> Unfortunately, there's one thing that's holding me back now: A pipeline
> containing the rest transform works fine when launched from the GUI using
> local or Beam-Direct. But it fails (see stack trace below) when launched
> via ./hop-run.sh using Beam-Direct and it always fails for DataFlow – which
> is what I want to use in the end.
>
> For Beam-Direct, I've verified the failure on both my macOS machine and on
> a debian bullseye VM, with these java versions:
>
> openjdk version "11.0.15" 2022-04-19
> OpenJDK Runtime Environment Homebrew (build 11.0.15+0)
> OpenJDK 64-Bit Server VM Homebrew (build 11.0.15+0, mixed mode)
>
> openjdk version "11.0.11" 2021-04-20
> OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)
> OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9,
> mixed mode)
>
> After some digging, my guess is that the rest transform implementation is
> based on jersey 1.19.4 but beam comes with jersey 2.3. I don't understand
> all the classloading magic going on, so for now I'm giving up.
>
> I'll create a ticket on Jira, but I still wanted to hear whether there's
> something like an easy way out. Ideas very much welcome!
>
> cheers
>
> Fabian
>
>
> 2022/07/13 13:39:16 - General - Executing this pipeline using the Beam
> Pipeline Engine with run configuration 'Beam-Direct'
> 2022/07/13 13:39:18 - GET PLT alerts.0 - ERROR: Error initializing
> transform [GET PLT alerts]
> 2022/07/13 13:39:18 - GET PLT alerts.0 - ERROR:
> java.lang.NoClassDefFoundError: javax/ws/rs/core/Link$Builder
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> java.base/java.lang.Class.getConstructor0(Class.java:3342)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> java.base/java.lang.Class.newInstance(Class.java:556)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:65)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:117)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:105)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> javax.ws.rs.core.MediaType.<clinit>(MediaType.java:44)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> org.apache.hop.pipeline.transforms.rest.Rest.init(Rest.java:571)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> org.apache.hop.pipeline.transform.TransformInitThread.run(TransformInitThread.java:66)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> java.base/java.lang.Thread.run(Thread.java:829)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - Caused by:
> java.lang.ClassNotFoundException: Unable to load class
> 'javax.ws.rs.core.Link$Builder' in this classloader or in the parent
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> org.apache.hop.core.plugins.HopURLClassLoader.loadClass(HopURLClassLoader.java:96)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - ... 12 more
> 2022/07/13 13:39:18 - GET PLT alerts.0 - Caused by:
> java.lang.ClassNotFoundException: javax.ws.rs.core.Link$Builder
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at java.base/java.net
> .URLClassLoader.findClass(URLClassLoader.java:471)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> org.apache.hop.core.plugins.HopURLClassLoader.loadClassFromThisLoader(HopURLClassLoader.java:66)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - at
> org.apache.hop.core.plugins.HopURLClassLoader.loadClass(HopURLClassLoader.java:91)
> 2022/07/13 13:39:18 - GET PLT alerts.0 - ... 13 more
> 2022/07/13 13:39:18 - ERROR: Transform [GET PLT alerts.0] failed to
> initialize!
> 2022/07/13 13:39:18 - General - ERROR: Error starting the Beam pipeline
> 2022/07/13 13:39:18 - General - ERROR:
> org.apache.hop.core.exception.HopException:
> 2022/07/13 13:39:18 - General - Error executing pipeline with runner Direct
> 2022/07/13 13:39:18 - General - java.lang.RuntimeException: Error
> executing TransformFn
> 2022/07/13 13:39:18 - General -
> 2022/07/13 13:39:18 - General - at
> org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:254)
> 2022/07/13 13:39:18 - General - at
> org.apache.hop.beam.engines.BeamPipelineEngine.lambda$startThreads$0(BeamPipelineEngine.java:301)
> 2022/07/13 13:39:18 - General - at
> java.base/java.lang.Thread.run(Thread.java:829)
> 2022/07/13 13:39:18 - General - Caused by:
> org.apache.beam.sdk.Pipeline$PipelineExecutionException:
> java.lang.RuntimeException: Error executing TransformFn
> 2022/07/13 13:39:18 - General - at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:373)
> 2022/07/13 13:39:18 - General - at
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:341)
> 2022/07/13 13:39:18 - General - at
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:218)
> 2022/07/13 13:39:18 - General - at
> org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:242)
> 2022/07/13 13:39:18 - General - ... 2 more
> 2022/07/13 13:39:18 - General - Caused by: java.lang.RuntimeException:
> Error executing TransformFn
> 2022/07/13 13:39:18 - General - at
> org.apache.hop.beam.core.transform.TransformTransform$TransformFn.processElement(TransformTransform.java:571)
> 2022/07/13 13:39:18 - General - Caused by:
> org.apache.hop.core.exception.HopException:
> 2022/07/13 13:39:18 - General - We failed to initialize at least one
> transform. Execution can not begin!
> 2022/07/13 13:39:18 - General -
> 2022/07/13 13:39:18 - General -
> 2022/07/13 13:39:18 - General - at
> org.apache.hop.pipeline.Pipeline.prepareExecution(Pipeline.java:1026)
> 2022/07/13 13:39:18 - General - at
> org.apache.hop.pipeline.engines.local.LocalPipelineEngine.prepareExecution(LocalPipelineEngine.java:206)
> 2022/07/13 13:39:18 - General - at
> org.apache.hop.beam.core.transform.TransformTransform$TransformFn.processElement(TransformTransform.java:441)
> 2022/07/13 13:39:18 - General - at
> org.apache.hop.beam.core.transform.TransformTransform$TransformFn$DoFnInvoker.invokeProcessElement(Unknown
> Source)
> 2022/07/13 13:39:18 - General - at
> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:228)
> 2022/07/13 13:39:18 - General - at
> org.apache.beam.repackaged.direct_java.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:184)
> 2022/07/13 13:39:18 - General - at
> org.apache.beam.repackaged.direct_java.runners.core.SimplePushbackSideInputDoFnRunner.processElementInReadyWindows(SimplePushbackSideInputDoFnRunner.java:79)
> 2022/07/13 13:39:18 - General - at
> org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:244)
> 2022/07/13 13:39:18 - General - at
> org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:54)
> 2022/07/13 13:39:18 - General - at
> org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:165)
> 2022/07/13 13:39:18 - General - at
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:129)
> 2022/07/13 13:39:18 - General - at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> 2022/07/13 13:39:18 - General - at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> 2022/07/13 13:39:18 - General - at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 2022/07/13 13:39:18 - General - at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 2022/07/13 13:39:18 - General - at
> java.base/java.lang.Thread.run(Thread.java:829)