You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Manas Kale <ma...@gmail.com> on 2020/08/21 06:16:30 UTC
AWS EMR deployment error : NoClassDefFoundError org/apache/flink/api/java/typeutils/ResultTypeQueryable
Hi,
I am trying to deploy a Flink jar on AWS EMR service. I have ensured that
Flink v1.10.0 is used in my pom file as that's the version supported by
EMR. However, I get the following error:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/flink/api/java/typeutils/ResultTypeQueryable
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.util.RunJar.run(RunJar.java:232)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.lang.ClassNotFoundException:
org.apache.flink.api.java.typeutils.ResultTypeQueryable
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 15 more
Also, if I deploy this on my local Flink cluster (v1.10.1) it works.
I'm not sure what could be the cause. Could it be because of misconfigured
classes bundled in the final JAR file or something that was patched in v
1.10.1?
Re: AWS EMR deployment error : NoClassDefFoundError org/apache/flink/api/java/typeutils/ResultTypeQueryable
Posted by Manas Kale <ma...@gmail.com>.
Thanks Prasanna and Chesnay. Changing the dependency scope worked and I
also had to add a maven shaded plugin transformer to resolve another error.
On Fri, Aug 21, 2020 at 11:38 PM Prasanna kumar <
prasannakumarramani@gmail.com> wrote:
> Manas,
>
> One option you could try is to set the scope in the dependencies as
> <scope>compile</scope> for the required artifacts rather than provided.
>
> Prasanna.
>
> On Fri, Aug 21, 2020 at 1:47 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> If this class cannot be found on the classpath then chances are Flink is
>> completely missing from the classpath.
>>
>> I haven't worked with EMR, but my guess is that you did not submit things
>> correctly.
>>
>> From the EMR documentation I could gather that the submission should work
>> without the submitted jar bundling all of Flink;
>>
>> given that you jar works in a local cluster that part should not be the
>> problem.
>>
>> On 21/08/2020 08:16, Manas Kale wrote:
>>
>> Hi,
>> I am trying to deploy a Flink jar on AWS EMR service. I have ensured that
>> Flink v1.10.0 is used in my pom file as that's the version supported by
>> EMR. However, I get the following error:
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/api/java/typeutils/ResultTypeQueryable
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
>> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:348)
>> at org.apache.hadoop.util.RunJar.run(RunJar.java:232)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
>> Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.java.typeutils.ResultTypeQueryable
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>> ... 15 more
>>
>> Also, if I deploy this on my local Flink cluster (v1.10.1) it works.
>> I'm not sure what could be the cause. Could it be because of
>> misconfigured classes bundled in the final JAR file or something that was
>> patched in v 1.10.1?
>>
>>
>>
Re: AWS EMR deployment error : NoClassDefFoundError org/apache/flink/api/java/typeutils/ResultTypeQueryable
Posted by Prasanna kumar <pr...@gmail.com>.
Manas,
One option you could try is to set the scope in the dependencies as <scope>
compile</scope> for the required artifacts rather than provided.
Prasanna.
On Fri, Aug 21, 2020 at 1:47 PM Chesnay Schepler <ch...@apache.org> wrote:
> If this class cannot be found on the classpath then chances are Flink is
> completely missing from the classpath.
>
> I haven't worked with EMR, but my guess is that you did not submit things
> correctly.
>
> From the EMR documentation I could gather that the submission should work
> without the submitted jar bundling all of Flink;
>
> given that you jar works in a local cluster that part should not be the
> problem.
>
> On 21/08/2020 08:16, Manas Kale wrote:
>
> Hi,
> I am trying to deploy a Flink jar on AWS EMR service. I have ensured that
> Flink v1.10.0 is used in my pom file as that's the version supported by
> EMR. However, I get the following error:
>
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/api/java/typeutils/ResultTypeQueryable
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:232)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.java.typeutils.ResultTypeQueryable
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 15 more
>
> Also, if I deploy this on my local Flink cluster (v1.10.1) it works.
> I'm not sure what could be the cause. Could it be because of misconfigured
> classes bundled in the final JAR file or something that was patched in v
> 1.10.1?
>
>
>
Re: AWS EMR deployment error : NoClassDefFoundError
org/apache/flink/api/java/typeutils/ResultTypeQueryable
Posted by Chesnay Schepler <ch...@apache.org>.
If this class cannot be found on the classpath then chances are Flink is
completely missing from the classpath.
I haven't worked with EMR, but my guess is that you did not submit
things correctly.
From the EMR documentation I could gather that the submission should
work without the submitted jar bundling all of Flink;
given that you jar works in a local cluster that part should not be the
problem.
On 21/08/2020 08:16, Manas Kale wrote:
> Hi,
> I am trying to deploy a Flink jar on AWS EMR service. I have ensured
> that Flink v1.10.0 is used in my pom file as that's the
> version supported by EMR. However, I get the following error:
> Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/api/java/typeutils/ResultTypeQueryable
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:232)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.java.typeutils.ResultTypeQueryable
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 15 more
> Also, if I deploy this on my local Flink cluster (v1.10.1) it works.
> I'm not sure what could be the cause. Could it be because of
> misconfigured classes bundled in the final JAR file or something that
> was patched in v 1.10.1?
>