You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Hanson, Bruce" <br...@here.com> on 2016/06/30 18:28:41 UTC
Error submitting stand-alone Flink job to EMR YARN cluster
I’m trying to submit a stand-alone Flink job to a YARN cluster running on EMR (Elastic MapReduce) nodes in AWS. When it tries to start a container for the Job Manager, it fails. The error message from the container is below. The command I’m using is:
$ flink run -m yarn-cluster -yn 1 -ynm test1:WordCount ./flink-1.0.3/examples/streaming/WordCount.jar
I have tried adding log4j and slf4j libraries to the classpath using -C and that doesn’t help.
This does not happen on other YARN clusters I have that are not EMR nodes. And it doesn’t happen on my EMR cluster if I use "yarn-session.sh" to create a Flink cluster in the YARN cluster and then use “flink run …” to submit the job to the Flink cluster.
Does anyone out there know how I could fix this?
Thanks in advance for any help you can give.
Error message in the jobmanager.err file:
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Level
at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:95)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.security.Groups.<init>(Groups.java:79)
at org.apache.hadoop.security.Groups.<init>(Groups.java:74)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
at org.apache.flink.yarn.ApplicationMasterBase.run(ApplicationMasterBase.scala:64)
at org.apache.flink.yarn.ApplicationMaster$.main(ApplicationMaster.scala:36)
at org.apache.flink.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Level
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 18 more
[cid:01830D03-C3D5-4C98-BB4D-27DC3889EBFC]
Bruce Hanson
Software Engineer
HERE Predictive Analytics
HERE Seattle
701 Pike St, Suite 2000, Seattle, WA 98101
47° 36' 41" N. 122° 19' 57" W<http://here.com/usa/seattle/98101/pike-st/701/map=47.611439,-122.332741,17/title=HERE%20Seattle%20-%20701%20Pike%20Street>
[cid:822C9F40-B358-441C-86A3-270D8B84D3CC]<http://360.here.com/> [cid:8C181B0D-7AD1-4C78-9D87-EB8B9235DE19] <https://twitter.com/here> [cid:6EF6C991-FACA-474E-8D17-3C91F7BD47C1] <https://www.facebook.com/here> [cid:53CE4C6B-3D8C-4E38-AE94-C7B2006A2BCE] <https://linkedin.com/company/heremaps> [cid:EA73988E-374C-4D16-A59C-AF82144FC71C] <https://www.instagram.com/here>
Re: Error submitting stand-alone Flink job to EMR YARN cluster
Posted by Jamie Grier <ja...@data-artisans.com>.
I know this is really basic but have you verified that you're Flink lib
folder contains log4j-1.2.17.jar? I imagine that's fine given the
yarn-session.sh approach is working fine. What version of EMR are you
running? What version of Flink?
-Jamie
On Thu, Jun 30, 2016 at 11:28 AM, Hanson, Bruce <br...@here.com>
wrote:
> I’m trying to submit a stand-alone Flink job to a YARN cluster running on
> EMR (Elastic MapReduce) nodes in AWS. When it tries to start a container
> for the Job Manager, it fails. The error message from the container is
> below. The command I’m using is:
>
> $ flink run -m yarn-cluster -yn 1 -ynm test1:WordCount
> ./flink-1.0.3/examples/streaming/WordCount.jar
>
> I have tried adding log4j and slf4j libraries to the classpath using -C
> and that doesn’t help.
>
> This does not happen on other YARN clusters I have that are not EMR nodes.
> And it doesn’t happen on my EMR cluster if I use "yarn-session.sh" to
> create a Flink cluster in the YARN cluster and then use “flink run …” to
> submit the job to the Flink cluster.
>
> Does anyone out there know how I could fix this?
>
> Thanks in advance for any help you can give.
>
> Error message in the jobmanager.err file:
>
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>
> SLF4J: Defaulting to no-operation (NOP) logger implementation
>
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/log4j/Level
>
> at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:357)
>
> at java.lang.Class.forName0(Native Method)
>
> at java.lang.Class.forName(Class.java:278)
>
> at
> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
>
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:95)
>
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
>
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
>
> at org.apache.hadoop.security.Groups.<init>(Groups.java:79)
>
> at org.apache.hadoop.security.Groups.<init>(Groups.java:74)
>
> at
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
>
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
>
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
>
> at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790)
>
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
>
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
>
> at
> org.apache.flink.yarn.ApplicationMasterBase.run(ApplicationMasterBase.scala:64)
>
> at
> org.apache.flink.yarn.ApplicationMaster$.main(ApplicationMaster.scala:36)
>
> at
> org.apache.flink.yarn.ApplicationMaster.main(ApplicationMaster.scala)
>
> Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Level
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
> ... 18 more
>
>
> *Bruce Hanson*
>
> Software Engineer
>
> HERE Predictive Analytics
>
>
>
> *HERE Seattle*
>
> 701 Pike St, Suite 2000, Seattle, WA 98101
>
> *47° 36' 41" N. 122° 19' 57" W
> <http://here.com/usa/seattle/98101/pike-st/701/map=47.611439,-122.332741,17/title=HERE%20Seattle%20-%20701%20Pike%20Street>*
>
>
>
> <http://360.here.com/> <https://twitter.com/here>
> <https://www.facebook.com/here> <https://linkedin.com/company/heremaps>
> <https://www.instagram.com/here>
>
>
>
>
--
Jamie Grier
data Artisans, Director of Applications Engineering
@jamiegrier <https://twitter.com/jamiegrier>
jamie@data-artisans.com
Re: Error submitting stand-alone Flink job to EMR YARN cluster
Posted by Ufuk Celebi <uc...@apache.org>.
You can also have a look at the YARN client logs, which should print which
JARs are uploaded. The container logs should also log the class path.
On Sun, Jul 3, 2016 at 6:04 PM, Jamie Grier <ja...@data-artisans.com> wrote:
> Hi Bruce,
>
> I just spun up an EMR cluster and tried this out. Hadoop 2.7.2 and Flink
> 1.0.3. I ran the exact same command as you and everything works just fine.
>
> Please verify one thing, though. In your command you do not specify the
> path to the Flink executable, which means it's just getting picked up from
> your PATH. Is it possible you're running a flink-1.0.3 example program but
> using a different version of Flink? Try the following:
>
> ./flink-1.0.3/bin/flink run -m yarn-cluster -yn 1 -ynm test1:WordCount ./flink-1.0.3/examples/streaming/WordCount.jar
>
>
>
> -Jamie
>
>
>
> On Thu, Jun 30, 2016 at 11:28 AM, Hanson, Bruce <br...@here.com>
> wrote:
>
>> I’m trying to submit a stand-alone Flink job to a YARN cluster running on
>> EMR (Elastic MapReduce) nodes in AWS. When it tries to start a container
>> for the Job Manager, it fails. The error message from the container is
>> below. The command I’m using is:
>>
>> $ flink run -m yarn-cluster -yn 1 -ynm test1:WordCount
>> ./flink-1.0.3/examples/streaming/WordCount.jar
>>
>> I have tried adding log4j and slf4j libraries to the classpath using -C
>> and that doesn’t help.
>>
>> This does not happen on other YARN clusters I have that are not EMR
>> nodes. And it doesn’t happen on my EMR cluster if I use "yarn-session.sh"
>> to create a Flink cluster in the YARN cluster and then use “flink run …” to
>> submit the job to the Flink cluster.
>>
>> Does anyone out there know how I could fix this?
>>
>> Thanks in advance for any help you can give.
>>
>> Error message in the jobmanager.err file:
>>
>> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>>
>> SLF4J: Defaulting to no-operation (NOP) logger implementation
>>
>> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
>> further details.
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/log4j/Level
>>
>> at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:357)
>>
>> at java.lang.Class.forName0(Native Method)
>>
>> at java.lang.Class.forName(Class.java:278)
>>
>> at
>> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
>>
>> at
>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:95)
>>
>> at
>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
>>
>> at
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
>>
>> at org.apache.hadoop.security.Groups.<init>(Groups.java:79)
>>
>> at org.apache.hadoop.security.Groups.<init>(Groups.java:74)
>>
>> at
>> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
>>
>> at
>> org.apache.flink.yarn.ApplicationMasterBase.run(ApplicationMasterBase.scala:64)
>>
>> at
>> org.apache.flink.yarn.ApplicationMaster$.main(ApplicationMaster.scala:36)
>>
>> at
>> org.apache.flink.yarn.ApplicationMaster.main(ApplicationMaster.scala)
>>
>> Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Level
>>
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>
>> ... 18 more
>>
>>
>> *Bruce Hanson*
>>
>> Software Engineer
>>
>> HERE Predictive Analytics
>>
>>
>>
>> *HERE Seattle*
>>
>> 701 Pike St, Suite 2000, Seattle, WA 98101
>>
>> *47° 36' 41" N. 122° 19' 57" W
>> <http://here.com/usa/seattle/98101/pike-st/701/map=47.611439,-122.332741,17/title=HERE%20Seattle%20-%20701%20Pike%20Street>*
>>
>>
>>
>> <http://360.here.com/> <https://twitter.com/here>
>> <https://www.facebook.com/here>
>> <https://linkedin.com/company/heremaps>
>> <https://www.instagram.com/here>
>>
>>
>>
>>
>
>
> --
>
> Jamie Grier
> data Artisans, Director of Applications Engineering
> @jamiegrier <https://twitter.com/jamiegrier>
> jamie@data-artisans.com
>
>
Re: Error submitting stand-alone Flink job to EMR YARN cluster
Posted by Jamie Grier <ja...@data-artisans.com>.
Hi Bruce,
I just spun up an EMR cluster and tried this out. Hadoop 2.7.2 and Flink
1.0.3. I ran the exact same command as you and everything works just fine.
Please verify one thing, though. In your command you do not specify the
path to the Flink executable, which means it's just getting picked up from
your PATH. Is it possible you're running a flink-1.0.3 example program but
using a different version of Flink? Try the following:
./flink-1.0.3/bin/flink run -m yarn-cluster -yn 1 -ynm test1:WordCount
./flink-1.0.3/examples/streaming/WordCount.jar
-Jamie
On Thu, Jun 30, 2016 at 11:28 AM, Hanson, Bruce <br...@here.com>
wrote:
> I’m trying to submit a stand-alone Flink job to a YARN cluster running on
> EMR (Elastic MapReduce) nodes in AWS. When it tries to start a container
> for the Job Manager, it fails. The error message from the container is
> below. The command I’m using is:
>
> $ flink run -m yarn-cluster -yn 1 -ynm test1:WordCount
> ./flink-1.0.3/examples/streaming/WordCount.jar
>
> I have tried adding log4j and slf4j libraries to the classpath using -C
> and that doesn’t help.
>
> This does not happen on other YARN clusters I have that are not EMR nodes.
> And it doesn’t happen on my EMR cluster if I use "yarn-session.sh" to
> create a Flink cluster in the YARN cluster and then use “flink run …” to
> submit the job to the Flink cluster.
>
> Does anyone out there know how I could fix this?
>
> Thanks in advance for any help you can give.
>
> Error message in the jobmanager.err file:
>
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
>
> SLF4J: Defaulting to no-operation (NOP) logger implementation
>
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/log4j/Level
>
> at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:357)
>
> at java.lang.Class.forName0(Native Method)
>
> at java.lang.Class.forName(Class.java:278)
>
> at
> org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
>
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:95)
>
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
>
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
>
> at org.apache.hadoop.security.Groups.<init>(Groups.java:79)
>
> at org.apache.hadoop.security.Groups.<init>(Groups.java:74)
>
> at
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
>
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
>
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
>
> at
> org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:790)
>
> at
> org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:760)
>
> at
> org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:633)
>
> at
> org.apache.flink.yarn.ApplicationMasterBase.run(ApplicationMasterBase.scala:64)
>
> at
> org.apache.flink.yarn.ApplicationMaster$.main(ApplicationMaster.scala:36)
>
> at
> org.apache.flink.yarn.ApplicationMaster.main(ApplicationMaster.scala)
>
> Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Level
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
> ... 18 more
>
>
> *Bruce Hanson*
>
> Software Engineer
>
> HERE Predictive Analytics
>
>
>
> *HERE Seattle*
>
> 701 Pike St, Suite 2000, Seattle, WA 98101
>
> *47° 36' 41" N. 122° 19' 57" W
> <http://here.com/usa/seattle/98101/pike-st/701/map=47.611439,-122.332741,17/title=HERE%20Seattle%20-%20701%20Pike%20Street>*
>
>
>
> <http://360.here.com/> <https://twitter.com/here>
> <https://www.facebook.com/here> <https://linkedin.com/company/heremaps>
> <https://www.instagram.com/here>
>
>
>
>
--
Jamie Grier
data Artisans, Director of Applications Engineering
@jamiegrier <https://twitter.com/jamiegrier>
jamie@data-artisans.com