You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Garvit Sharma <ga...@gmail.com> on 2018/06/13 17:03:02 UTC
Exception while submitting jobs through Yarn
Hi,
I am using *flink-1.5.0-bin-hadoop27-scala_2.11 *to submit jobs through
Yarn, but I am getting the below exception :
java.lang.NoClassDefFoundError:
com/sun/jersey/core/util/FeaturesAndProperties
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.getClusterDescriptor(FlinkYarnSessionCli.java:966)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.createDescriptor(FlinkYarnSessionCli.java:269)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:444)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:92)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:221)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
Caused by: java.lang.ClassNotFoundException:
com.sun.jersey.core.util.FeaturesAndProperties
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
Command : HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -m yarn-cluster
-yd -yn 2 -ys 20 -yjm 10240 -ytm 10240 -yst -ynm test -yqu default -p 20
test.jar
The class *com/sun/jersey/core/util/FeaturesAndProperties* is already
present in the test.jar so not sure why am I getting this exception.
Please check and let me know.
Thanks,
--
Garvit Sharma
github.com/garvitlnmiit/
No Body is a Scholar by birth, its only hard work and strong determination
that makes him master.
Re: Exception while submitting jobs through Yarn
Posted by Till Rohrmann <tr...@apache.org>.
Great to hear. Please open a PR for the improvements you like to contribute.
Cheers,
Till
On Wed, Jun 20, 2018 at 4:56 PM Garvit Sharma <ga...@gmail.com> wrote:
> So, finally, I have got this working. The issue was because of a poor
> library which was using xerces 2.6 :).
>
> In this process, I found few things missing from the doc would like to
> contribute the same.
>
> I really appreciate the support provided.
>
> Thanks,
>
> On Tue, 19 Jun 2018 at 4:05 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Since you're using a vendor's distro, I would suggest asking on their
>> user forum.
>>
>> Cheers
>>
>> -------- Original message --------
>> From: Garvit Sharma <ga...@gmail.com>
>> Date: 6/19/18 3:34 AM (GMT-08:00)
>> To: trohrmann@apache.org
>> Cc: Amit Jain <aj...@gmail.com>, Chesnay Schepler <ch...@apache.org>,
>> Ted Yu <yu...@gmail.com>, user@flink.apache.org
>> Subject: Re: Exception while submitting jobs through Yarn
>>
>> Any help on this?
>>
>> On Mon, Jun 18, 2018 at 11:31 PM Garvit Sharma <ga...@gmail.com>
>> wrote:
>>
>>> Yes, it is.
>>>
>>> On Mon, Jun 18, 2018 at 7:54 PM Till Rohrmann <tr...@apache.org>
>>> wrote:
>>>
>>>> Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to `
>>>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`?
>>>>
>>>> On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma <ga...@gmail.com>
>>>> wrote:
>>>>
>>>>> I don't think I can access core-default as it comes with Hadoop jar
>>>>>
>>>>> On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann <tr...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hmm, could you check whether core-default.xml contains any suspicious
>>>>>> entries? Apparently xerces:2.9.1 cannot read it.
>>>>>>
>>>>>> On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <ga...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> After putting the following log in my code, I can see that the
>>>>>>> Xerces version is - Xerces version : Xerces-J 2.9.1
>>>>>>>
>>>>>>> log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());
>>>>>>>
>>>>>>> Also, following is the response of *$* *locate xerces* command on
>>>>>>> the server -
>>>>>>>
>>>>>>>
>>>>>>> /usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar
>>>>>>>
>>>>>>>
>>>>>>> /usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar
>>>>>>>
>>>>>>>
>>>>>>> /usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar
>>>>>>>
>>>>>>>
>>>>>>> /usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar
>>>>>>>
>>>>>>> /usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar
>>>>>>>
>>>>>>>
Re: Exception while submitting jobs through Yarn
Posted by Garvit Sharma <ga...@gmail.com>.
So, finally, I have got this working. The issue was because of a poor
library which was using xerces 2.6 :).
In this process, I found few things missing from the doc would like to
contribute the same.
I really appreciate the support provided.
Thanks,
On Tue, 19 Jun 2018 at 4:05 PM, Ted Yu <yu...@gmail.com> wrote:
> Since you're using a vendor's distro, I would suggest asking on their user
> forum.
>
> Cheers
>
> -------- Original message --------
> From: Garvit Sharma <ga...@gmail.com>
> Date: 6/19/18 3:34 AM (GMT-08:00)
> To: trohrmann@apache.org
> Cc: Amit Jain <aj...@gmail.com>, Chesnay Schepler <ch...@apache.org>,
> Ted Yu <yu...@gmail.com>, user@flink.apache.org
> Subject: Re: Exception while submitting jobs through Yarn
>
> Any help on this?
>
> On Mon, Jun 18, 2018 at 11:31 PM Garvit Sharma <ga...@gmail.com>
> wrote:
>
>> Yes, it is.
>>
>> On Mon, Jun 18, 2018 at 7:54 PM Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>>> Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to `
>>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`?
>>>
>>> On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma <ga...@gmail.com>
>>> wrote:
>>>
>>>> I don't think I can access core-default as it comes with Hadoop jar
>>>>
>>>> On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann <tr...@apache.org>
>>>> wrote:
>>>>
>>>>> Hmm, could you check whether core-default.xml contains any suspicious
>>>>> entries? Apparently xerces:2.9.1 cannot read it.
>>>>>
>>>>> On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <ga...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> After putting the following log in my code, I can see that the Xerces
>>>>>> version is - Xerces version : Xerces-J 2.9.1
>>>>>>
>>>>>> log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());
>>>>>>
>>>>>> Also, following is the response of *$* *locate xerces* command on
>>>>>> the server -
>>>>>>
>>>>>>
>>>>>> /usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar
>>>>>>
>>>>>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar
>>>>>>
>>>>>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar
>>>>>>
>>>>>> /usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>>>>>>
>>>>>> /usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar
>>>>>>
>>>>>>
>>>>>> /usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>>>>>>
>>>>>> /usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar
>>>>>>
>>>>>> /usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar
>>>>>>
>>>>>> /usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar
>>>>>>
>>>>>> /usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar
>>>>>>
>>>>>> /usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar
>>>>>>
>>>>>> /usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar
>>>>>>
>>>>>> /usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar
>>>>>>
>>>>>>
>>>>>> /usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
>>>>>>
>>>>>> /usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar
>>>>>>
>>>>>> /usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar
>>>>>>
>>>>>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar
>>>>>>
>>>>>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar
>>>>>>
>>>>>> /usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>>>>>>
>>>>>> /usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar
>>>>>>
>>>>>>
>>>>>> /usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>>>>>>
>>>>>> /usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar
>>>>>>
>>>>>> /usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar
>>>>>>
>>>>>> /usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar
>>>>>>
>>>>>>
Re: Exception while submitting jobs through Yarn
Posted by Ted Yu <yu...@gmail.com>.
Since you're using a vendor's distro, I would suggest asking on their user forum.
Cheers
-------- Original message --------From: Garvit Sharma <ga...@gmail.com> Date: 6/19/18 3:34 AM (GMT-08:00) To: trohrmann@apache.org Cc: Amit Jain <aj...@gmail.com>, Chesnay Schepler <ch...@apache.org>, Ted Yu <yu...@gmail.com>, user@flink.apache.org Subject: Re: Exception while submitting jobs through Yarn
Any help on this?
On Mon, Jun 18, 2018 at 11:31 PM Garvit Sharma <ga...@gmail.com> wrote:
Yes, it is.
On Mon, Jun 18, 2018 at 7:54 PM Till Rohrmann <tr...@apache.org> wrote:
Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`?
On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma <ga...@gmail.com> wrote:
I don't think I can access core-default as it comes with Hadoop jar
On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann <tr...@apache.org> wrote:
Hmm, could you check whether core-default.xml contains any suspicious entries? Apparently xerces:2.9.1 cannot read it.
On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <ga...@gmail.com> wrote:
Hi,
After putting the following log in my code, I can see that the Xerces version is - Xerces version : Xerces-J 2.9.1log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());Also, following is the response of $ locate xerces command on the server -
/usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar
/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar
/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar
/usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar
/usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar
/usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar
/usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar
/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar
/usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar
Re: Exception while submitting jobs through Yarn
Posted by Garvit Sharma <ga...@gmail.com>.
Any help on this?
On Mon, Jun 18, 2018 at 11:31 PM Garvit Sharma <ga...@gmail.com> wrote:
> Yes, it is.
>
> On Mon, Jun 18, 2018 at 7:54 PM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to `
>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`?
>>
>> On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma <ga...@gmail.com>
>> wrote:
>>
>>> I don't think I can access core-default as it comes with Hadoop jar
>>>
>>> On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann <tr...@apache.org>
>>> wrote:
>>>
>>>> Hmm, could you check whether core-default.xml contains any suspicious
>>>> entries? Apparently xerces:2.9.1 cannot read it.
>>>>
>>>> On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <ga...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> After putting the following log in my code, I can see that the Xerces
>>>>> version is - Xerces version : Xerces-J 2.9.1
>>>>>
>>>>> log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());
>>>>>
>>>>> Also, following is the response of *$* *locate xerces* command on the
>>>>> server -
>>>>>
>>>>>
>>>>> /usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar
>>>>>
>>>>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar
>>>>>
>>>>> /usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar
>>>>>
>>>>>
>>>>> /usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar
>>>>>
>>>>> /usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar
>>>>>
>>>>> /usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar
>>>>>
>>>>> /usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar
>>>>>
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/oozie/libserver/xercesImpl-2.10.0.jar
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/oozie/libtools/xercesImpl-2.10.0.jar
>>>>>
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/ranger-admin/ews/webapp/WEB-INF/lib/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/slider/lib/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/spark2/jars/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
>>>>>
>>>>> /usr/hdp/2.6.3.0-235/zookeeper/lib/xercesMinimal-1.9.6.2.jar
>>>>>
>>>>> /usr/hdp/share/hst/hst-common/lib/xercesImpl-2.9.1.jar
>>>>>
>>>>> Now, I can say that the version of xerces are same.
>>>>>
>>>>>
>>>>> So, what is causing this issue if Xerces version is in sync?
>>>>>
>>>>>
>>>>> I am very excited to discover the issue :)
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> On Mon, Jun 18, 2018 at 6:27 PM Till Rohrmann <tr...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Could you check which xerces version you have on your classpath?
>>>>>> Apparently, it cannot read core-default.xml as Ted pointed out. This might
>>>>>> be the root cause for the failure.
>>>>>>
>>>>>> Cheers,
>>>>>> Till
>>>>>>
>>>>>> On Mon, Jun 18, 2018 at 1:31 PM Garvit Sharma <ga...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Sorry for the confusion, but the yarn is running on Hadoop version
>>>>>>> 2.7 only and hence I am using Flink 1.5 Hadoop 2.7 binary.
>>>>>>>
>>>>>>> Below are the details provided by Yarn version command :
>>>>>>>
>>>>>>> Hadoop 2.7.3.2.6.3.0-235
>>>>>>> Subversion git@github.com:hortonworks/hadoop.git -r
>>>>>>> 45bfd33bba8acadfa0e6024c80981c023b28d454
>>>>>>> Compiled by jenkins on 2017-10-30T02:31Z
>>>>>>> Compiled with protoc 2.5.0
>>>>>>> From source with checksum cd1a4a466ef450f547c279989f3aa3
>>>>>>> This command was run using
>>>>>>> /usr/hdp/2.6.3.0-235/hadoop/hadoop-common-2.7.3.2.6.3.0-235.jar
>>>>>>>
>>>>>>> Please let me know if you have found the resolution to my issue :)
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jun 18, 2018 at 4:50 PM Till Rohrmann <tr...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Which Hadoop version have you installed? It looks as if Flink has
>>>>>>>> been build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class
>>>>>>>> path. If you want to run Flink on Hadoop 2.6, then try to use the Hadoop
>>>>>>>> free Flink binaries or the one built for Hadoop 2.6.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Till
>>>>>>>>
>>>>>>>> On Mon, Jun 18, 2018 at 10:48 AM Garvit Sharma <ga...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Ok, I have attached the log file.
>>>>>>>>>
>>>>>>>>> Please check and let me know.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <aj...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Gravit,
>>>>>>>>>>
>>>>>>>>>> I think Till is interested to know about classpath details
>>>>>>>>>> present at the start of JM and TM logs e.g. following logs provide
>>>>>>>>>> classpath details used by TM in our case.
>>>>>>>>>>
>>>>>>>>>> 2018-06-17 19:01:30,656 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>>> --------------------------------------------------------------------------------
>>>>>>>>>> 2018-06-17 19:01:30,658 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Starting
>>>>>>>>>> YARN TaskExecutor runner (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @
>>>>>>>>>> 14:54:44 UTC)
>>>>>>>>>> 2018-06-17 19:01:30,659 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - OS
>>>>>>>>>> current user: yarn
>>>>>>>>>> 2018-06-17 19:01:31,662 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Current
>>>>>>>>>> Hadoop/Kerberos user: hadoop
>>>>>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM:
>>>>>>>>>> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10
>>>>>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Maximum
>>>>>>>>>> heap size: 6647 MiBytes
>>>>>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>>> JAVA_HOME: /usr/lib/jvm/java-openjdk
>>>>>>>>>> 2018-06-17 19:01:31,664 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Hadoop
>>>>>>>>>> version: 2.8.3
>>>>>>>>>> 2018-06-17 19:01:31,664 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM
>>>>>>>>>> Options:
>>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>>> -Xms6936m
>>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>>> -Xmx6936m
>>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>>> -XX:MaxDirectMemorySize=4072m
>>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>>> -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log
>>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>>> -Dlogback.configurationFile=file:./logback.xml
>>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>>> -Dlog4j.configuration=file:./log4j.properties
>>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Program
>>>>>>>>>> Arguments:
>>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>>> --configDir
>>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - .
>>>>>>>>>> *2018-06-17 19:01:31,666 INFO
>>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>>> Classpath:
>>>>>>>>>> lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........*
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Thanks,
>>>>>>>>>> Amit
>>>>>>>>>>
>>>>>>>>>> On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <
>>>>>>>>>> garvits45@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Please refer to my previous mail for complete logs.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <
>>>>>>>>>>> trohrmann@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Could you also please share the complete log file with us.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Till
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yu...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> The error for core-default.xml is interesting.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Flink doesn't have this file. Probably it came with Yarn.
>>>>>>>>>>>>> Please check the hadoop version Flink was built with versus the hadoop
>>>>>>>>>>>>> version in your cluster.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>> -------- Original message --------
>>>>>>>>>>>>> From: Garvit Sharma <ga...@gmail.com>
>>>>>>>>>>>>> Date: 6/16/18 7:23 AM (GMT-08:00)
>>>>>>>>>>>>> To: trohrmann@apache.org
>>>>>>>>>>>>> Cc: Chesnay Schepler <ch...@apache.org>,
>>>>>>>>>>>>> user@flink.apache.org
>>>>>>>>>>>>> Subject: Re: Exception while submitting jobs through Yarn
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am not able to figure out, got stuck badly in this since
>>>>>>>>>>>>> last 1 week. Any little help would be appreciated.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2018-06-16 19:25:10,523 DEBUG
>>>>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator
>>>>>>>>>>>>> - Parallelism set: 1 for 8
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2018-06-16 19:25:10,578 DEBUG
>>>>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator
>>>>>>>>>>>>> - Parallelism set: 1 for 1
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2018-06-16 19:25:10,588 DEBUG
>>>>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator
>>>>>>>>>>>>> - CONNECTED: KeyGroupStreamPartitioner - 1 -> 8
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2018-06-16 19:25:10,591 DEBUG
>>>>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator
>>>>>>>>>>>>> - Parallelism set: 1 for 5
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2018-06-16 19:25:10,597 DEBUG
>>>>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator
>>>>>>>>>>>>> - CONNECTED: KeyGroupStreamPartitioner - 5 -> 8
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2018-06-16 19:25:10,618 FATAL
>>>>>>>>>>>>> org.apache.hadoop.conf.Configuration -
>>>>>>>>>>>>> error parsing conf core-default.xml
>>>>>>>>>>>>>
>>>>>>>>>>>>> javax.xml.parsers.ParserConfigurationException: Feature '
>>>>>>>>>>>>> http://apache.org/xml/features/xinclude' is not recognized.
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>>>>>>>>>>>>> Source)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2018-06-16 19:25:10,620 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>>>>>>>>>>>> - Error while getting queue information from YARN:
>>>>>>>>>>>>> null
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2018-06-16 19:25:10,621 DEBUG
>>>>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor -
>>>>>>>>>>>>> Error details
>>>>>>>>>>>>>
>>>>>>>>>>>>> java.lang.ExceptionInInitializerError
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>>>>>>>
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Garvit Sharma
>>>>>>>>>>> github.com/garvitlnmiit/
>>>>>>>>>>>
>>>>>>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>>>>>>> determination that makes him master.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Garvit Sharma
>>>>>>>>> github.com/garvitlnmiit/
>>>>>>>>>
>>>>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>>>>> determination that makes him master.
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Garvit Sharma
>>>>>>> github.com/garvitlnmiit/
>>>>>>>
>>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>>> determination that makes him master.
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Garvit Sharma
>>>>> github.com/garvitlnmiit/
>>>>>
>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>> determination that makes him master.
>>>>>
>>>> --
>>>
>>> Garvit Sharma
>>> github.com/garvitlnmiit/
>>>
>>> No Body is a Scholar by birth, its only hard work and strong
>>> determination that makes him master.
>>>
>>
>
> --
>
> Garvit Sharma
> github.com/garvitlnmiit/
>
> No Body is a Scholar by birth, its only hard work and strong determination
> that makes him master.
>
--
Garvit Sharma
github.com/garvitlnmiit/
No Body is a Scholar by birth, its only hard work and strong determination
that makes him master.
Re: Exception while submitting jobs through Yarn
Posted by Garvit Sharma <ga...@gmail.com>.
Yes, it is.
On Mon, Jun 18, 2018 at 7:54 PM Till Rohrmann <tr...@apache.org> wrote:
> Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to `
> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`?
>
> On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma <ga...@gmail.com> wrote:
>
>> I don't think I can access core-default as it comes with Hadoop jar
>>
>> On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>>> Hmm, could you check whether core-default.xml contains any suspicious
>>> entries? Apparently xerces:2.9.1 cannot read it.
>>>
>>> On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <ga...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> After putting the following log in my code, I can see that the Xerces
>>>> version is - Xerces version : Xerces-J 2.9.1
>>>>
>>>> log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());
>>>>
>>>> Also, following is the response of *$* *locate xerces* command on the
>>>> server -
>>>>
>>>>
>>>> /usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar
>>>>
>>>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar
>>>>
>>>> /usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar
>>>>
>>>>
>>>> /usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar
>>>>
>>>> /usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar
>>>>
>>>> /usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar
>>>>
>>>> /usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar
>>>>
>>>> /usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar
>>>>
>>>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar
>>>>
>>>> /usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar
>>>>
>>>>
>>>> /usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar
>>>>
>>>> /usr/hdp/2.6.3.0-235/oozie/libserver/xercesImpl-2.10.0.jar
>>>>
>>>> /usr/hdp/2.6.3.0-235/oozie/libtools/xercesImpl-2.10.0.jar
>>>>
>>>>
>>>> /usr/hdp/2.6.3.0-235/ranger-admin/ews/webapp/WEB-INF/lib/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.3.0-235/slider/lib/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.3.0-235/spark2/jars/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.3.0-235/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
>>>>
>>>> /usr/hdp/2.6.3.0-235/zookeeper/lib/xercesMinimal-1.9.6.2.jar
>>>>
>>>> /usr/hdp/share/hst/hst-common/lib/xercesImpl-2.9.1.jar
>>>>
>>>> Now, I can say that the version of xerces are same.
>>>>
>>>>
>>>> So, what is causing this issue if Xerces version is in sync?
>>>>
>>>>
>>>> I am very excited to discover the issue :)
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> On Mon, Jun 18, 2018 at 6:27 PM Till Rohrmann <tr...@apache.org>
>>>> wrote:
>>>>
>>>>> Could you check which xerces version you have on your classpath?
>>>>> Apparently, it cannot read core-default.xml as Ted pointed out. This might
>>>>> be the root cause for the failure.
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On Mon, Jun 18, 2018 at 1:31 PM Garvit Sharma <ga...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Sorry for the confusion, but the yarn is running on Hadoop version
>>>>>> 2.7 only and hence I am using Flink 1.5 Hadoop 2.7 binary.
>>>>>>
>>>>>> Below are the details provided by Yarn version command :
>>>>>>
>>>>>> Hadoop 2.7.3.2.6.3.0-235
>>>>>> Subversion git@github.com:hortonworks/hadoop.git -r
>>>>>> 45bfd33bba8acadfa0e6024c80981c023b28d454
>>>>>> Compiled by jenkins on 2017-10-30T02:31Z
>>>>>> Compiled with protoc 2.5.0
>>>>>> From source with checksum cd1a4a466ef450f547c279989f3aa3
>>>>>> This command was run using
>>>>>> /usr/hdp/2.6.3.0-235/hadoop/hadoop-common-2.7.3.2.6.3.0-235.jar
>>>>>>
>>>>>> Please let me know if you have found the resolution to my issue :)
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>
>>>>>> On Mon, Jun 18, 2018 at 4:50 PM Till Rohrmann <tr...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Which Hadoop version have you installed? It looks as if Flink has
>>>>>>> been build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class
>>>>>>> path. If you want to run Flink on Hadoop 2.6, then try to use the Hadoop
>>>>>>> free Flink binaries or the one built for Hadoop 2.6.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Till
>>>>>>>
>>>>>>> On Mon, Jun 18, 2018 at 10:48 AM Garvit Sharma <ga...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Ok, I have attached the log file.
>>>>>>>>
>>>>>>>> Please check and let me know.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <aj...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Gravit,
>>>>>>>>>
>>>>>>>>> I think Till is interested to know about classpath details present
>>>>>>>>> at the start of JM and TM logs e.g. following logs provide classpath
>>>>>>>>> details used by TM in our case.
>>>>>>>>>
>>>>>>>>> 2018-06-17 19:01:30,656 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>> --------------------------------------------------------------------------------
>>>>>>>>> 2018-06-17 19:01:30,658 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Starting
>>>>>>>>> YARN TaskExecutor runner (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @
>>>>>>>>> 14:54:44 UTC)
>>>>>>>>> 2018-06-17 19:01:30,659 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - OS
>>>>>>>>> current user: yarn
>>>>>>>>> 2018-06-17 19:01:31,662 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Current
>>>>>>>>> Hadoop/Kerberos user: hadoop
>>>>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM:
>>>>>>>>> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10
>>>>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Maximum
>>>>>>>>> heap size: 6647 MiBytes
>>>>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>> JAVA_HOME: /usr/lib/jvm/java-openjdk
>>>>>>>>> 2018-06-17 19:01:31,664 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Hadoop
>>>>>>>>> version: 2.8.3
>>>>>>>>> 2018-06-17 19:01:31,664 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM
>>>>>>>>> Options:
>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>> -Xms6936m
>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>> -Xmx6936m
>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>> -XX:MaxDirectMemorySize=4072m
>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>> -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log
>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>> -Dlogback.configurationFile=file:./logback.xml
>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>> -Dlog4j.configuration=file:./log4j.properties
>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Program
>>>>>>>>> Arguments:
>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>> --configDir
>>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - .
>>>>>>>>> *2018-06-17 19:01:31,666 INFO
>>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>>> Classpath:
>>>>>>>>> lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........*
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Thanks,
>>>>>>>>> Amit
>>>>>>>>>
>>>>>>>>> On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <
>>>>>>>>> garvits45@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Please refer to my previous mail for complete logs.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <
>>>>>>>>>> trohrmann@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Could you also please share the complete log file with us.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Till
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yu...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> The error for core-default.xml is interesting.
>>>>>>>>>>>>
>>>>>>>>>>>> Flink doesn't have this file. Probably it came with Yarn.
>>>>>>>>>>>> Please check the hadoop version Flink was built with versus the hadoop
>>>>>>>>>>>> version in your cluster.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>> -------- Original message --------
>>>>>>>>>>>> From: Garvit Sharma <ga...@gmail.com>
>>>>>>>>>>>> Date: 6/16/18 7:23 AM (GMT-08:00)
>>>>>>>>>>>> To: trohrmann@apache.org
>>>>>>>>>>>> Cc: Chesnay Schepler <ch...@apache.org>,
>>>>>>>>>>>> user@flink.apache.org
>>>>>>>>>>>> Subject: Re: Exception while submitting jobs through Yarn
>>>>>>>>>>>>
>>>>>>>>>>>> I am not able to figure out, got stuck badly in this since last
>>>>>>>>>>>> 1 week. Any little help would be appreciated.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2018-06-16 19:25:10,523 DEBUG
>>>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator
>>>>>>>>>>>> - Parallelism set: 1 for 8
>>>>>>>>>>>>
>>>>>>>>>>>> 2018-06-16 19:25:10,578 DEBUG
>>>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator
>>>>>>>>>>>> - Parallelism set: 1 for 1
>>>>>>>>>>>>
>>>>>>>>>>>> 2018-06-16 19:25:10,588 DEBUG
>>>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator
>>>>>>>>>>>> - CONNECTED: KeyGroupStreamPartitioner - 1 -> 8
>>>>>>>>>>>>
>>>>>>>>>>>> 2018-06-16 19:25:10,591 DEBUG
>>>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator
>>>>>>>>>>>> - Parallelism set: 1 for 5
>>>>>>>>>>>>
>>>>>>>>>>>> 2018-06-16 19:25:10,597 DEBUG
>>>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator
>>>>>>>>>>>> - CONNECTED: KeyGroupStreamPartitioner - 5 -> 8
>>>>>>>>>>>>
>>>>>>>>>>>> 2018-06-16 19:25:10,618 FATAL
>>>>>>>>>>>> org.apache.hadoop.conf.Configuration -
>>>>>>>>>>>> error parsing conf core-default.xml
>>>>>>>>>>>>
>>>>>>>>>>>> javax.xml.parsers.ParserConfigurationException: Feature '
>>>>>>>>>>>> http://apache.org/xml/features/xinclude' is not recognized.
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>>>>>>>>>>>> Source)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>>>>>>>>>>>
>>>>>>>>>>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>>>>>>
>>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>>>>>>
>>>>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>>>>>>>>>
>>>>>>>>>>>> 2018-06-16 19:25:10,620 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>>>>>>>>>>> - Error while getting queue information from YARN:
>>>>>>>>>>>> null
>>>>>>>>>>>>
>>>>>>>>>>>> 2018-06-16 19:25:10,621 DEBUG
>>>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor -
>>>>>>>>>>>> Error details
>>>>>>>>>>>>
>>>>>>>>>>>> java.lang.ExceptionInInitializerError
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Garvit Sharma
>>>>>>>>>> github.com/garvitlnmiit/
>>>>>>>>>>
>>>>>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>>>>>> determination that makes him master.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Garvit Sharma
>>>>>>>> github.com/garvitlnmiit/
>>>>>>>>
>>>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>>>> determination that makes him master.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Garvit Sharma
>>>>>> github.com/garvitlnmiit/
>>>>>>
>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>> determination that makes him master.
>>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Garvit Sharma
>>>> github.com/garvitlnmiit/
>>>>
>>>> No Body is a Scholar by birth, its only hard work and strong
>>>> determination that makes him master.
>>>>
>>> --
>>
>> Garvit Sharma
>> github.com/garvitlnmiit/
>>
>> No Body is a Scholar by birth, its only hard work and strong
>> determination that makes him master.
>>
>
--
Garvit Sharma
github.com/garvitlnmiit/
No Body is a Scholar by birth, its only hard work and strong determination
that makes him master.
Re: Exception while submitting jobs through Yarn
Posted by Till Rohrmann <tr...@apache.org>.
Is `/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar` a link to `
/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar`?
On Mon, Jun 18, 2018 at 4:02 PM Garvit Sharma <ga...@gmail.com> wrote:
> I don't think I can access core-default as it comes with Hadoop jar
>
> On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Hmm, could you check whether core-default.xml contains any suspicious
>> entries? Apparently xerces:2.9.1 cannot read it.
>>
>> On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <ga...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> After putting the following log in my code, I can see that the Xerces
>>> version is - Xerces version : Xerces-J 2.9.1
>>>
>>> log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());
>>>
>>> Also, following is the response of *$* *locate xerces* command on the
>>> server -
>>>
>>>
>>> /usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar
>>>
>>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar
>>>
>>> /usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar
>>>
>>>
>>> /usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar
>>>
>>> /usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar
>>>
>>> /usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar
>>>
>>> /usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar
>>>
>>> /usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar
>>>
>>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar
>>>
>>> /usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar
>>>
>>>
>>> /usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar
>>>
>>> /usr/hdp/2.6.3.0-235/oozie/libserver/xercesImpl-2.10.0.jar
>>>
>>> /usr/hdp/2.6.3.0-235/oozie/libtools/xercesImpl-2.10.0.jar
>>>
>>>
>>> /usr/hdp/2.6.3.0-235/ranger-admin/ews/webapp/WEB-INF/lib/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.3.0-235/slider/lib/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.3.0-235/spark2/jars/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.3.0-235/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
>>>
>>> /usr/hdp/2.6.3.0-235/zookeeper/lib/xercesMinimal-1.9.6.2.jar
>>>
>>> /usr/hdp/share/hst/hst-common/lib/xercesImpl-2.9.1.jar
>>>
>>> Now, I can say that the version of xerces are same.
>>>
>>>
>>> So, what is causing this issue if Xerces version is in sync?
>>>
>>>
>>> I am very excited to discover the issue :)
>>>
>>>
>>> Thanks,
>>>
>>> On Mon, Jun 18, 2018 at 6:27 PM Till Rohrmann <tr...@apache.org>
>>> wrote:
>>>
>>>> Could you check which xerces version you have on your classpath?
>>>> Apparently, it cannot read core-default.xml as Ted pointed out. This might
>>>> be the root cause for the failure.
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On Mon, Jun 18, 2018 at 1:31 PM Garvit Sharma <ga...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Sorry for the confusion, but the yarn is running on Hadoop version 2.7
>>>>> only and hence I am using Flink 1.5 Hadoop 2.7 binary.
>>>>>
>>>>> Below are the details provided by Yarn version command :
>>>>>
>>>>> Hadoop 2.7.3.2.6.3.0-235
>>>>> Subversion git@github.com:hortonworks/hadoop.git -r
>>>>> 45bfd33bba8acadfa0e6024c80981c023b28d454
>>>>> Compiled by jenkins on 2017-10-30T02:31Z
>>>>> Compiled with protoc 2.5.0
>>>>> From source with checksum cd1a4a466ef450f547c279989f3aa3
>>>>> This command was run using
>>>>> /usr/hdp/2.6.3.0-235/hadoop/hadoop-common-2.7.3.2.6.3.0-235.jar
>>>>>
>>>>> Please let me know if you have found the resolution to my issue :)
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>> On Mon, Jun 18, 2018 at 4:50 PM Till Rohrmann <tr...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Which Hadoop version have you installed? It looks as if Flink has
>>>>>> been build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class
>>>>>> path. If you want to run Flink on Hadoop 2.6, then try to use the Hadoop
>>>>>> free Flink binaries or the one built for Hadoop 2.6.
>>>>>>
>>>>>> Cheers,
>>>>>> Till
>>>>>>
>>>>>> On Mon, Jun 18, 2018 at 10:48 AM Garvit Sharma <ga...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Ok, I have attached the log file.
>>>>>>>
>>>>>>> Please check and let me know.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <aj...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Gravit,
>>>>>>>>
>>>>>>>> I think Till is interested to know about classpath details present
>>>>>>>> at the start of JM and TM logs e.g. following logs provide classpath
>>>>>>>> details used by TM in our case.
>>>>>>>>
>>>>>>>> 2018-06-17 19:01:30,656 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>> --------------------------------------------------------------------------------
>>>>>>>> 2018-06-17 19:01:30,658 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Starting
>>>>>>>> YARN TaskExecutor runner (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @
>>>>>>>> 14:54:44 UTC)
>>>>>>>> 2018-06-17 19:01:30,659 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - OS
>>>>>>>> current user: yarn
>>>>>>>> 2018-06-17 19:01:31,662 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Current
>>>>>>>> Hadoop/Kerberos user: hadoop
>>>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM:
>>>>>>>> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10
>>>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Maximum
>>>>>>>> heap size: 6647 MiBytes
>>>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>> JAVA_HOME: /usr/lib/jvm/java-openjdk
>>>>>>>> 2018-06-17 19:01:31,664 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Hadoop
>>>>>>>> version: 2.8.3
>>>>>>>> 2018-06-17 19:01:31,664 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM
>>>>>>>> Options:
>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>> -Xms6936m
>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>> -Xmx6936m
>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>> -XX:MaxDirectMemorySize=4072m
>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>> -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log
>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>> -Dlogback.configurationFile=file:./logback.xml
>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>> -Dlog4j.configuration=file:./log4j.properties
>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Program
>>>>>>>> Arguments:
>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>> --configDir
>>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - .
>>>>>>>> *2018-06-17 19:01:31,666 INFO
>>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>>> Classpath:
>>>>>>>> lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........*
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks,
>>>>>>>> Amit
>>>>>>>>
>>>>>>>> On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <garvits45@gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Please refer to my previous mail for complete logs.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <
>>>>>>>>> trohrmann@apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> Could you also please share the complete log file with us.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Till
>>>>>>>>>>
>>>>>>>>>> On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yu...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> The error for core-default.xml is interesting.
>>>>>>>>>>>
>>>>>>>>>>> Flink doesn't have this file. Probably it came with Yarn. Please
>>>>>>>>>>> check the hadoop version Flink was built with versus the hadoop version in
>>>>>>>>>>> your cluster.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> -------- Original message --------
>>>>>>>>>>> From: Garvit Sharma <ga...@gmail.com>
>>>>>>>>>>> Date: 6/16/18 7:23 AM (GMT-08:00)
>>>>>>>>>>> To: trohrmann@apache.org
>>>>>>>>>>> Cc: Chesnay Schepler <ch...@apache.org>, user@flink.apache.org
>>>>>>>>>>> Subject: Re: Exception while submitting jobs through Yarn
>>>>>>>>>>>
>>>>>>>>>>> I am not able to figure out, got stuck badly in this since last
>>>>>>>>>>> 1 week. Any little help would be appreciated.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2018-06-16 19:25:10,523 DEBUG
>>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator
>>>>>>>>>>> - Parallelism set: 1 for 8
>>>>>>>>>>>
>>>>>>>>>>> 2018-06-16 19:25:10,578 DEBUG
>>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator
>>>>>>>>>>> - Parallelism set: 1 for 1
>>>>>>>>>>>
>>>>>>>>>>> 2018-06-16 19:25:10,588 DEBUG
>>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator
>>>>>>>>>>> - CONNECTED: KeyGroupStreamPartitioner - 1 -> 8
>>>>>>>>>>>
>>>>>>>>>>> 2018-06-16 19:25:10,591 DEBUG
>>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator
>>>>>>>>>>> - Parallelism set: 1 for 5
>>>>>>>>>>>
>>>>>>>>>>> 2018-06-16 19:25:10,597 DEBUG
>>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator
>>>>>>>>>>> - CONNECTED: KeyGroupStreamPartitioner - 5 -> 8
>>>>>>>>>>>
>>>>>>>>>>> 2018-06-16 19:25:10,618 FATAL
>>>>>>>>>>> org.apache.hadoop.conf.Configuration -
>>>>>>>>>>> error parsing conf core-default.xml
>>>>>>>>>>>
>>>>>>>>>>> javax.xml.parsers.ParserConfigurationException: Feature '
>>>>>>>>>>> http://apache.org/xml/features/xinclude' is not recognized.
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>>>>>>>>>>> Source)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>>>>>>>>>>
>>>>>>>>>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>>>>>
>>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>>>>>
>>>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>>>>>>>>
>>>>>>>>>>> 2018-06-16 19:25:10,620 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>>>>>>>>>> - Error while getting queue information from YARN:
>>>>>>>>>>> null
>>>>>>>>>>>
>>>>>>>>>>> 2018-06-16 19:25:10,621 DEBUG
>>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor -
>>>>>>>>>>> Error details
>>>>>>>>>>>
>>>>>>>>>>> java.lang.ExceptionInInitializerError
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Garvit Sharma
>>>>>>>>> github.com/garvitlnmiit/
>>>>>>>>>
>>>>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>>>>> determination that makes him master.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Garvit Sharma
>>>>>>> github.com/garvitlnmiit/
>>>>>>>
>>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>>> determination that makes him master.
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Garvit Sharma
>>>>> github.com/garvitlnmiit/
>>>>>
>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>> determination that makes him master.
>>>>>
>>>>
>>>
>>> --
>>>
>>> Garvit Sharma
>>> github.com/garvitlnmiit/
>>>
>>> No Body is a Scholar by birth, its only hard work and strong
>>> determination that makes him master.
>>>
>> --
>
> Garvit Sharma
> github.com/garvitlnmiit/
>
> No Body is a Scholar by birth, its only hard work and strong determination
> that makes him master.
>
Re: Exception while submitting jobs through Yarn
Posted by Garvit Sharma <ga...@gmail.com>.
I don't think I can access core-default as it comes with Hadoop jar
On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann <tr...@apache.org> wrote:
> Hmm, could you check whether core-default.xml contains any suspicious
> entries? Apparently xerces:2.9.1 cannot read it.
>
> On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <ga...@gmail.com> wrote:
>
>> Hi,
>>
>> After putting the following log in my code, I can see that the Xerces
>> version is - Xerces version : Xerces-J 2.9.1
>>
>> log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());
>>
>> Also, following is the response of *$* *locate xerces* command on the
>> server -
>>
>>
>> /usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar
>>
>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar
>>
>> /usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar
>>
>>
>> /usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar
>>
>> /usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar
>>
>> /usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar
>>
>> /usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar
>>
>> /usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar
>>
>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar
>>
>> /usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar
>>
>>
>> /usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar
>>
>> /usr/hdp/2.6.3.0-235/oozie/libserver/xercesImpl-2.10.0.jar
>>
>> /usr/hdp/2.6.3.0-235/oozie/libtools/xercesImpl-2.10.0.jar
>>
>>
>> /usr/hdp/2.6.3.0-235/ranger-admin/ews/webapp/WEB-INF/lib/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/slider/lib/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/spark2/jars/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/zookeeper/lib/xercesMinimal-1.9.6.2.jar
>>
>> /usr/hdp/share/hst/hst-common/lib/xercesImpl-2.9.1.jar
>>
>> Now, I can say that the version of xerces are same.
>>
>>
>> So, what is causing this issue if Xerces version is in sync?
>>
>>
>> I am very excited to discover the issue :)
>>
>>
>> Thanks,
>>
>> On Mon, Jun 18, 2018 at 6:27 PM Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>>> Could you check which xerces version you have on your classpath?
>>> Apparently, it cannot read core-default.xml as Ted pointed out. This might
>>> be the root cause for the failure.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Mon, Jun 18, 2018 at 1:31 PM Garvit Sharma <ga...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Sorry for the confusion, but the yarn is running on Hadoop version 2.7
>>>> only and hence I am using Flink 1.5 Hadoop 2.7 binary.
>>>>
>>>> Below are the details provided by Yarn version command :
>>>>
>>>> Hadoop 2.7.3.2.6.3.0-235
>>>> Subversion git@github.com:hortonworks/hadoop.git -r
>>>> 45bfd33bba8acadfa0e6024c80981c023b28d454
>>>> Compiled by jenkins on 2017-10-30T02:31Z
>>>> Compiled with protoc 2.5.0
>>>> From source with checksum cd1a4a466ef450f547c279989f3aa3
>>>> This command was run using
>>>> /usr/hdp/2.6.3.0-235/hadoop/hadoop-common-2.7.3.2.6.3.0-235.jar
>>>>
>>>> Please let me know if you have found the resolution to my issue :)
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> On Mon, Jun 18, 2018 at 4:50 PM Till Rohrmann <tr...@apache.org>
>>>> wrote:
>>>>
>>>>> Which Hadoop version have you installed? It looks as if Flink has been
>>>>> build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class path. If
>>>>> you want to run Flink on Hadoop 2.6, then try to use the Hadoop free Flink
>>>>> binaries or the one built for Hadoop 2.6.
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On Mon, Jun 18, 2018 at 10:48 AM Garvit Sharma <ga...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Ok, I have attached the log file.
>>>>>>
>>>>>> Please check and let me know.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <aj...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Gravit,
>>>>>>>
>>>>>>> I think Till is interested to know about classpath details present
>>>>>>> at the start of JM and TM logs e.g. following logs provide classpath
>>>>>>> details used by TM in our case.
>>>>>>>
>>>>>>> 2018-06-17 19:01:30,656 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> --------------------------------------------------------------------------------
>>>>>>> 2018-06-17 19:01:30,658 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Starting
>>>>>>> YARN TaskExecutor runner (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @
>>>>>>> 14:54:44 UTC)
>>>>>>> 2018-06-17 19:01:30,659 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - OS
>>>>>>> current user: yarn
>>>>>>> 2018-06-17 19:01:31,662 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Current
>>>>>>> Hadoop/Kerberos user: hadoop
>>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM:
>>>>>>> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10
>>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Maximum
>>>>>>> heap size: 6647 MiBytes
>>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> JAVA_HOME: /usr/lib/jvm/java-openjdk
>>>>>>> 2018-06-17 19:01:31,664 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Hadoop
>>>>>>> version: 2.8.3
>>>>>>> 2018-06-17 19:01:31,664 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM
>>>>>>> Options:
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> -Xms6936m
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> -Xmx6936m
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> -XX:MaxDirectMemorySize=4072m
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> -Dlogback.configurationFile=file:./logback.xml
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> -Dlog4j.configuration=file:./log4j.properties
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Program
>>>>>>> Arguments:
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> --configDir
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - .
>>>>>>> *2018-06-17 19:01:31,666 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> Classpath:
>>>>>>> lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........*
>>>>>>>
>>>>>>> --
>>>>>>> Thanks,
>>>>>>> Amit
>>>>>>>
>>>>>>> On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <ga...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Please refer to my previous mail for complete logs.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <tr...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Could you also please share the complete log file with us.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Till
>>>>>>>>>
>>>>>>>>> On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yu...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> The error for core-default.xml is interesting.
>>>>>>>>>>
>>>>>>>>>> Flink doesn't have this file. Probably it came with Yarn. Please
>>>>>>>>>> check the hadoop version Flink was built with versus the hadoop version in
>>>>>>>>>> your cluster.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> -------- Original message --------
>>>>>>>>>> From: Garvit Sharma <ga...@gmail.com>
>>>>>>>>>> Date: 6/16/18 7:23 AM (GMT-08:00)
>>>>>>>>>> To: trohrmann@apache.org
>>>>>>>>>> Cc: Chesnay Schepler <ch...@apache.org>, user@flink.apache.org
>>>>>>>>>> Subject: Re: Exception while submitting jobs through Yarn
>>>>>>>>>>
>>>>>>>>>> I am not able to figure out, got stuck badly in this since last 1
>>>>>>>>>> week. Any little help would be appreciated.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,523 DEBUG
>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>>>> Parallelism set: 1 for 8
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,578 DEBUG
>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>>>> Parallelism set: 1 for 1
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,588 DEBUG
>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>>>> CONNECTED: KeyGroupStreamPartitioner - 1 -> 8
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,591 DEBUG
>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>>>> Parallelism set: 1 for 5
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,597 DEBUG
>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>>>> CONNECTED: KeyGroupStreamPartitioner - 5 -> 8
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration
>>>>>>>>>> - error parsing conf core-default.xml
>>>>>>>>>>
>>>>>>>>>> javax.xml.parsers.ParserConfigurationException: Feature '
>>>>>>>>>> http://apache.org/xml/features/xinclude' is not recognized.
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>>>>>>>>>> Source)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>>>>>>>>>
>>>>>>>>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>>>>
>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>>>>
>>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,620 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>>>>>>>>> - Error while getting queue information from YARN: null
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,621 DEBUG
>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor -
>>>>>>>>>> Error details
>>>>>>>>>>
>>>>>>>>>> java.lang.ExceptionInInitializerError
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Garvit Sharma
>>>>>>>> github.com/garvitlnmiit/
>>>>>>>>
>>>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>>>> determination that makes him master.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Garvit Sharma
>>>>>> github.com/garvitlnmiit/
>>>>>>
>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>> determination that makes him master.
>>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Garvit Sharma
>>>> github.com/garvitlnmiit/
>>>>
>>>> No Body is a Scholar by birth, its only hard work and strong
>>>> determination that makes him master.
>>>>
>>>
>>
>> --
>>
>> Garvit Sharma
>> github.com/garvitlnmiit/
>>
>> No Body is a Scholar by birth, its only hard work and strong
>> determination that makes him master.
>>
> --
Garvit Sharma
github.com/garvitlnmiit/
No Body is a Scholar by birth, its only hard work and strong determination
that makes him master.
Re: Exception while submitting jobs through Yarn
Posted by Till Rohrmann <tr...@apache.org>.
Hmm, could you check whether core-default.xml contains any suspicious
entries? Apparently xerces:2.9.1 cannot read it.
On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <ga...@gmail.com> wrote:
> Hi,
>
> After putting the following log in my code, I can see that the Xerces
> version is - Xerces version : Xerces-J 2.9.1
>
> log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());
>
> Also, following is the response of *$* *locate xerces* command on the
> server -
>
>
> /usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar
>
> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar
>
> /usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar
>
>
> /usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar
>
> /usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar
>
> /usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar
>
> /usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar
>
> /usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar
>
> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar
>
> /usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar
>
>
> /usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar
>
> /usr/hdp/2.6.3.0-235/oozie/libserver/xercesImpl-2.10.0.jar
>
> /usr/hdp/2.6.3.0-235/oozie/libtools/xercesImpl-2.10.0.jar
>
>
> /usr/hdp/2.6.3.0-235/ranger-admin/ews/webapp/WEB-INF/lib/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.3.0-235/slider/lib/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.3.0-235/spark2/jars/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.3.0-235/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
>
> /usr/hdp/2.6.3.0-235/zookeeper/lib/xercesMinimal-1.9.6.2.jar
>
> /usr/hdp/share/hst/hst-common/lib/xercesImpl-2.9.1.jar
>
> Now, I can say that the version of xerces are same.
>
>
> So, what is causing this issue if Xerces version is in sync?
>
>
> I am very excited to discover the issue :)
>
>
> Thanks,
>
> On Mon, Jun 18, 2018 at 6:27 PM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Could you check which xerces version you have on your classpath?
>> Apparently, it cannot read core-default.xml as Ted pointed out. This might
>> be the root cause for the failure.
>>
>> Cheers,
>> Till
>>
>> On Mon, Jun 18, 2018 at 1:31 PM Garvit Sharma <ga...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Sorry for the confusion, but the yarn is running on Hadoop version 2.7
>>> only and hence I am using Flink 1.5 Hadoop 2.7 binary.
>>>
>>> Below are the details provided by Yarn version command :
>>>
>>> Hadoop 2.7.3.2.6.3.0-235
>>> Subversion git@github.com:hortonworks/hadoop.git -r
>>> 45bfd33bba8acadfa0e6024c80981c023b28d454
>>> Compiled by jenkins on 2017-10-30T02:31Z
>>> Compiled with protoc 2.5.0
>>> From source with checksum cd1a4a466ef450f547c279989f3aa3
>>> This command was run using
>>> /usr/hdp/2.6.3.0-235/hadoop/hadoop-common-2.7.3.2.6.3.0-235.jar
>>>
>>> Please let me know if you have found the resolution to my issue :)
>>>
>>> Thanks,
>>>
>>>
>>> On Mon, Jun 18, 2018 at 4:50 PM Till Rohrmann <tr...@apache.org>
>>> wrote:
>>>
>>>> Which Hadoop version have you installed? It looks as if Flink has been
>>>> build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class path. If
>>>> you want to run Flink on Hadoop 2.6, then try to use the Hadoop free Flink
>>>> binaries or the one built for Hadoop 2.6.
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On Mon, Jun 18, 2018 at 10:48 AM Garvit Sharma <ga...@gmail.com>
>>>> wrote:
>>>>
>>>>> Ok, I have attached the log file.
>>>>>
>>>>> Please check and let me know.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <aj...@gmail.com> wrote:
>>>>>
>>>>>> Hi Gravit,
>>>>>>
>>>>>> I think Till is interested to know about classpath details present at
>>>>>> the start of JM and TM logs e.g. following logs provide classpath details
>>>>>> used by TM in our case.
>>>>>>
>>>>>> 2018-06-17 19:01:30,656 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>> --------------------------------------------------------------------------------
>>>>>> 2018-06-17 19:01:30,658 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Starting
>>>>>> YARN TaskExecutor runner (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @
>>>>>> 14:54:44 UTC)
>>>>>> 2018-06-17 19:01:30,659 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - OS
>>>>>> current user: yarn
>>>>>> 2018-06-17 19:01:31,662 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Current
>>>>>> Hadoop/Kerberos user: hadoop
>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM:
>>>>>> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10
>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Maximum
>>>>>> heap size: 6647 MiBytes
>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>> JAVA_HOME: /usr/lib/jvm/java-openjdk
>>>>>> 2018-06-17 19:01:31,664 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Hadoop
>>>>>> version: 2.8.3
>>>>>> 2018-06-17 19:01:31,664 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM
>>>>>> Options:
>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>> -Xms6936m
>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>> -Xmx6936m
>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>> -XX:MaxDirectMemorySize=4072m
>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>> -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log
>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>> -Dlogback.configurationFile=file:./logback.xml
>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>> -Dlog4j.configuration=file:./log4j.properties
>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Program
>>>>>> Arguments:
>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>> --configDir
>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - .
>>>>>> *2018-06-17 19:01:31,666 INFO
>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>> Classpath:
>>>>>> lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........*
>>>>>>
>>>>>> --
>>>>>> Thanks,
>>>>>> Amit
>>>>>>
>>>>>> On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <ga...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Please refer to my previous mail for complete logs.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <tr...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Could you also please share the complete log file with us.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Till
>>>>>>>>
>>>>>>>> On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yu...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> The error for core-default.xml is interesting.
>>>>>>>>>
>>>>>>>>> Flink doesn't have this file. Probably it came with Yarn. Please
>>>>>>>>> check the hadoop version Flink was built with versus the hadoop version in
>>>>>>>>> your cluster.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> -------- Original message --------
>>>>>>>>> From: Garvit Sharma <ga...@gmail.com>
>>>>>>>>> Date: 6/16/18 7:23 AM (GMT-08:00)
>>>>>>>>> To: trohrmann@apache.org
>>>>>>>>> Cc: Chesnay Schepler <ch...@apache.org>, user@flink.apache.org
>>>>>>>>> Subject: Re: Exception while submitting jobs through Yarn
>>>>>>>>>
>>>>>>>>> I am not able to figure out, got stuck badly in this since last 1
>>>>>>>>> week. Any little help would be appreciated.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2018-06-16 19:25:10,523 DEBUG
>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>>> Parallelism set: 1 for 8
>>>>>>>>>
>>>>>>>>> 2018-06-16 19:25:10,578 DEBUG
>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>>> Parallelism set: 1 for 1
>>>>>>>>>
>>>>>>>>> 2018-06-16 19:25:10,588 DEBUG
>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>>> CONNECTED: KeyGroupStreamPartitioner - 1 -> 8
>>>>>>>>>
>>>>>>>>> 2018-06-16 19:25:10,591 DEBUG
>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>>> Parallelism set: 1 for 5
>>>>>>>>>
>>>>>>>>> 2018-06-16 19:25:10,597 DEBUG
>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>>> CONNECTED: KeyGroupStreamPartitioner - 5 -> 8
>>>>>>>>>
>>>>>>>>> 2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration
>>>>>>>>> - error parsing conf core-default.xml
>>>>>>>>>
>>>>>>>>> javax.xml.parsers.ParserConfigurationException: Feature '
>>>>>>>>> http://apache.org/xml/features/xinclude' is not recognized.
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>>>>>>>>> Source)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>>>>>>>>
>>>>>>>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>>>
>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>>>
>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>>>>>>
>>>>>>>>> 2018-06-16 19:25:10,620 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>>>>>>>> - Error while getting queue information from YARN: null
>>>>>>>>>
>>>>>>>>> 2018-06-16 19:25:10,621 DEBUG
>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor -
>>>>>>>>> Error details
>>>>>>>>>
>>>>>>>>> java.lang.ExceptionInInitializerError
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Garvit Sharma
>>>>>>> github.com/garvitlnmiit/
>>>>>>>
>>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>>> determination that makes him master.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Garvit Sharma
>>>>> github.com/garvitlnmiit/
>>>>>
>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>> determination that makes him master.
>>>>>
>>>>
>>>
>>> --
>>>
>>> Garvit Sharma
>>> github.com/garvitlnmiit/
>>>
>>> No Body is a Scholar by birth, its only hard work and strong
>>> determination that makes him master.
>>>
>>
>
> --
>
> Garvit Sharma
> github.com/garvitlnmiit/
>
> No Body is a Scholar by birth, its only hard work and strong determination
> that makes him master.
>
Re: Exception while submitting jobs through Yarn
Posted by Garvit Sharma <ga...@gmail.com>.
Hi,
After putting the following log in my code, I can see that the Xerces
version is - Xerces version : Xerces-J 2.9.1
log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());
Also, following is the response of *$* *locate xerces* command on the
server -
/usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar
/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar
/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar
/usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar
/usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar
/usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
/usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar
/usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar
/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar
/usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar
/usr/hdp/2.6.3.0-235/oozie/libserver/xercesImpl-2.10.0.jar
/usr/hdp/2.6.3.0-235/oozie/libtools/xercesImpl-2.10.0.jar
/usr/hdp/2.6.3.0-235/ranger-admin/ews/webapp/WEB-INF/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/slider/lib/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/spark2/jars/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
/usr/hdp/2.6.3.0-235/zookeeper/lib/xercesMinimal-1.9.6.2.jar
/usr/hdp/share/hst/hst-common/lib/xercesImpl-2.9.1.jar
Now, I can say that the version of xerces are same.
So, what is causing this issue if Xerces version is in sync?
I am very excited to discover the issue :)
Thanks,
On Mon, Jun 18, 2018 at 6:27 PM Till Rohrmann <tr...@apache.org> wrote:
> Could you check which xerces version you have on your classpath?
> Apparently, it cannot read core-default.xml as Ted pointed out. This might
> be the root cause for the failure.
>
> Cheers,
> Till
>
> On Mon, Jun 18, 2018 at 1:31 PM Garvit Sharma <ga...@gmail.com> wrote:
>
>> Hi,
>>
>> Sorry for the confusion, but the yarn is running on Hadoop version 2.7
>> only and hence I am using Flink 1.5 Hadoop 2.7 binary.
>>
>> Below are the details provided by Yarn version command :
>>
>> Hadoop 2.7.3.2.6.3.0-235
>> Subversion git@github.com:hortonworks/hadoop.git -r
>> 45bfd33bba8acadfa0e6024c80981c023b28d454
>> Compiled by jenkins on 2017-10-30T02:31Z
>> Compiled with protoc 2.5.0
>> From source with checksum cd1a4a466ef450f547c279989f3aa3
>> This command was run using
>> /usr/hdp/2.6.3.0-235/hadoop/hadoop-common-2.7.3.2.6.3.0-235.jar
>>
>> Please let me know if you have found the resolution to my issue :)
>>
>> Thanks,
>>
>>
>> On Mon, Jun 18, 2018 at 4:50 PM Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>>> Which Hadoop version have you installed? It looks as if Flink has been
>>> build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class path. If
>>> you want to run Flink on Hadoop 2.6, then try to use the Hadoop free Flink
>>> binaries or the one built for Hadoop 2.6.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Mon, Jun 18, 2018 at 10:48 AM Garvit Sharma <ga...@gmail.com>
>>> wrote:
>>>
>>>> Ok, I have attached the log file.
>>>>
>>>> Please check and let me know.
>>>>
>>>> Thanks,
>>>>
>>>> On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <aj...@gmail.com> wrote:
>>>>
>>>>> Hi Gravit,
>>>>>
>>>>> I think Till is interested to know about classpath details present at
>>>>> the start of JM and TM logs e.g. following logs provide classpath details
>>>>> used by TM in our case.
>>>>>
>>>>> 2018-06-17 19:01:30,656 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>> --------------------------------------------------------------------------------
>>>>> 2018-06-17 19:01:30,658 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Starting
>>>>> YARN TaskExecutor runner (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @
>>>>> 14:54:44 UTC)
>>>>> 2018-06-17 19:01:30,659 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - OS
>>>>> current user: yarn
>>>>> 2018-06-17 19:01:31,662 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Current
>>>>> Hadoop/Kerberos user: hadoop
>>>>> 2018-06-17 19:01:31,663 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM:
>>>>> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10
>>>>> 2018-06-17 19:01:31,663 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Maximum
>>>>> heap size: 6647 MiBytes
>>>>> 2018-06-17 19:01:31,663 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>> JAVA_HOME: /usr/lib/jvm/java-openjdk
>>>>> 2018-06-17 19:01:31,664 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Hadoop
>>>>> version: 2.8.3
>>>>> 2018-06-17 19:01:31,664 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM
>>>>> Options:
>>>>> 2018-06-17 19:01:31,665 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>> -Xms6936m
>>>>> 2018-06-17 19:01:31,665 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>> -Xmx6936m
>>>>> 2018-06-17 19:01:31,665 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>> -XX:MaxDirectMemorySize=4072m
>>>>> 2018-06-17 19:01:31,665 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>> -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log
>>>>> 2018-06-17 19:01:31,665 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>> -Dlogback.configurationFile=file:./logback.xml
>>>>> 2018-06-17 19:01:31,665 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>> -Dlog4j.configuration=file:./log4j.properties
>>>>> 2018-06-17 19:01:31,665 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Program
>>>>> Arguments:
>>>>> 2018-06-17 19:01:31,665 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>> --configDir
>>>>> 2018-06-17 19:01:31,665 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - .
>>>>> *2018-06-17 19:01:31,666 INFO
>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>> Classpath:
>>>>> lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........*
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Amit
>>>>>
>>>>> On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <ga...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Please refer to my previous mail for complete logs.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <tr...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Could you also please share the complete log file with us.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Till
>>>>>>>
>>>>>>> On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yu...@gmail.com> wrote:
>>>>>>>
>>>>>>>> The error for core-default.xml is interesting.
>>>>>>>>
>>>>>>>> Flink doesn't have this file. Probably it came with Yarn. Please
>>>>>>>> check the hadoop version Flink was built with versus the hadoop version in
>>>>>>>> your cluster.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> -------- Original message --------
>>>>>>>> From: Garvit Sharma <ga...@gmail.com>
>>>>>>>> Date: 6/16/18 7:23 AM (GMT-08:00)
>>>>>>>> To: trohrmann@apache.org
>>>>>>>> Cc: Chesnay Schepler <ch...@apache.org>, user@flink.apache.org
>>>>>>>> Subject: Re: Exception while submitting jobs through Yarn
>>>>>>>>
>>>>>>>> I am not able to figure out, got stuck badly in this since last 1
>>>>>>>> week. Any little help would be appreciated.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2018-06-16 19:25:10,523 DEBUG
>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>> Parallelism set: 1 for 8
>>>>>>>>
>>>>>>>> 2018-06-16 19:25:10,578 DEBUG
>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>> Parallelism set: 1 for 1
>>>>>>>>
>>>>>>>> 2018-06-16 19:25:10,588 DEBUG
>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>> CONNECTED: KeyGroupStreamPartitioner - 1 -> 8
>>>>>>>>
>>>>>>>> 2018-06-16 19:25:10,591 DEBUG
>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>> Parallelism set: 1 for 5
>>>>>>>>
>>>>>>>> 2018-06-16 19:25:10,597 DEBUG
>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>> CONNECTED: KeyGroupStreamPartitioner - 5 -> 8
>>>>>>>>
>>>>>>>> 2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration
>>>>>>>> - error parsing conf core-default.xml
>>>>>>>>
>>>>>>>> javax.xml.parsers.ParserConfigurationException: Feature '
>>>>>>>> http://apache.org/xml/features/xinclude' is not recognized.
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>>>>>>>> Source)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
>>>>>>>>
>>>>>>>> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>>>>>>>
>>>>>>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>>
>>>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>>
>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>>
>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>>>>>
>>>>>>>> 2018-06-16 19:25:10,620 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>>>>>>> - Error while getting queue information from YARN: null
>>>>>>>>
>>>>>>>> 2018-06-16 19:25:10,621 DEBUG
>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor -
>>>>>>>> Error details
>>>>>>>>
>>>>>>>> java.lang.ExceptionInInitializerError
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>>
>>>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Garvit Sharma
>>>>>> github.com/garvitlnmiit/
>>>>>>
>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>> determination that makes him master.
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Garvit Sharma
>>>> github.com/garvitlnmiit/
>>>>
>>>> No Body is a Scholar by birth, its only hard work and strong
>>>> determination that makes him master.
>>>>
>>>
>>
>> --
>>
>> Garvit Sharma
>> github.com/garvitlnmiit/
>>
>> No Body is a Scholar by birth, its only hard work and strong
>> determination that makes him master.
>>
>
--
Garvit Sharma
github.com/garvitlnmiit/
No Body is a Scholar by birth, its only hard work and strong determination
that makes him master.
Re: Exception while submitting jobs through Yarn
Posted by Till Rohrmann <tr...@apache.org>.
Could you check which xerces version you have on your classpath?
Apparently, it cannot read core-default.xml as Ted pointed out. This might
be the root cause for the failure.
Cheers,
Till
On Mon, Jun 18, 2018 at 1:31 PM Garvit Sharma <ga...@gmail.com> wrote:
> Hi,
>
> Sorry for the confusion, but the yarn is running on Hadoop version 2.7
> only and hence I am using Flink 1.5 Hadoop 2.7 binary.
>
> Below are the details provided by Yarn version command :
>
> Hadoop 2.7.3.2.6.3.0-235
> Subversion git@github.com:hortonworks/hadoop.git -r
> 45bfd33bba8acadfa0e6024c80981c023b28d454
> Compiled by jenkins on 2017-10-30T02:31Z
> Compiled with protoc 2.5.0
> From source with checksum cd1a4a466ef450f547c279989f3aa3
> This command was run using
> /usr/hdp/2.6.3.0-235/hadoop/hadoop-common-2.7.3.2.6.3.0-235.jar
>
> Please let me know if you have found the resolution to my issue :)
>
> Thanks,
>
>
> On Mon, Jun 18, 2018 at 4:50 PM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Which Hadoop version have you installed? It looks as if Flink has been
>> build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class path. If
>> you want to run Flink on Hadoop 2.6, then try to use the Hadoop free Flink
>> binaries or the one built for Hadoop 2.6.
>>
>> Cheers,
>> Till
>>
>> On Mon, Jun 18, 2018 at 10:48 AM Garvit Sharma <ga...@gmail.com>
>> wrote:
>>
>>> Ok, I have attached the log file.
>>>
>>> Please check and let me know.
>>>
>>> Thanks,
>>>
>>> On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <aj...@gmail.com> wrote:
>>>
>>>> Hi Gravit,
>>>>
>>>> I think Till is interested to know about classpath details present at
>>>> the start of JM and TM logs e.g. following logs provide classpath details
>>>> used by TM in our case.
>>>>
>>>> 2018-06-17 19:01:30,656 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>> --------------------------------------------------------------------------------
>>>> 2018-06-17 19:01:30,658 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Starting
>>>> YARN TaskExecutor runner (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @
>>>> 14:54:44 UTC)
>>>> 2018-06-17 19:01:30,659 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - OS
>>>> current user: yarn
>>>> 2018-06-17 19:01:31,662 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Current
>>>> Hadoop/Kerberos user: hadoop
>>>> 2018-06-17 19:01:31,663 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM:
>>>> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10
>>>> 2018-06-17 19:01:31,663 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Maximum
>>>> heap size: 6647 MiBytes
>>>> 2018-06-17 19:01:31,663 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>> JAVA_HOME: /usr/lib/jvm/java-openjdk
>>>> 2018-06-17 19:01:31,664 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Hadoop
>>>> version: 2.8.3
>>>> 2018-06-17 19:01:31,664 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM
>>>> Options:
>>>> 2018-06-17 19:01:31,665 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>> -Xms6936m
>>>> 2018-06-17 19:01:31,665 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>> -Xmx6936m
>>>> 2018-06-17 19:01:31,665 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>> -XX:MaxDirectMemorySize=4072m
>>>> 2018-06-17 19:01:31,665 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>> -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log
>>>> 2018-06-17 19:01:31,665 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>> -Dlogback.configurationFile=file:./logback.xml
>>>> 2018-06-17 19:01:31,665 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>> -Dlog4j.configuration=file:./log4j.properties
>>>> 2018-06-17 19:01:31,665 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Program
>>>> Arguments:
>>>> 2018-06-17 19:01:31,665 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>> --configDir
>>>> 2018-06-17 19:01:31,665 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - .
>>>> *2018-06-17 19:01:31,666 INFO
>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>> Classpath:
>>>> lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........*
>>>>
>>>> --
>>>> Thanks,
>>>> Amit
>>>>
>>>> On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <ga...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Please refer to my previous mail for complete logs.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <tr...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Could you also please share the complete log file with us.
>>>>>>
>>>>>> Cheers,
>>>>>> Till
>>>>>>
>>>>>> On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> The error for core-default.xml is interesting.
>>>>>>>
>>>>>>> Flink doesn't have this file. Probably it came with Yarn. Please
>>>>>>> check the hadoop version Flink was built with versus the hadoop version in
>>>>>>> your cluster.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> -------- Original message --------
>>>>>>> From: Garvit Sharma <ga...@gmail.com>
>>>>>>> Date: 6/16/18 7:23 AM (GMT-08:00)
>>>>>>> To: trohrmann@apache.org
>>>>>>> Cc: Chesnay Schepler <ch...@apache.org>, user@flink.apache.org
>>>>>>> Subject: Re: Exception while submitting jobs through Yarn
>>>>>>>
>>>>>>> I am not able to figure out, got stuck badly in this since last 1
>>>>>>> week. Any little help would be appreciated.
>>>>>>>
>>>>>>>
>>>>>>> 2018-06-16 19:25:10,523 DEBUG
>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>> Parallelism set: 1 for 8
>>>>>>>
>>>>>>> 2018-06-16 19:25:10,578 DEBUG
>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>> Parallelism set: 1 for 1
>>>>>>>
>>>>>>> 2018-06-16 19:25:10,588 DEBUG
>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>> CONNECTED: KeyGroupStreamPartitioner - 1 -> 8
>>>>>>>
>>>>>>> 2018-06-16 19:25:10,591 DEBUG
>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>> Parallelism set: 1 for 5
>>>>>>>
>>>>>>> 2018-06-16 19:25:10,597 DEBUG
>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>> CONNECTED: KeyGroupStreamPartitioner - 5 -> 8
>>>>>>>
>>>>>>> 2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration
>>>>>>> - error parsing conf core-default.xml
>>>>>>>
>>>>>>> javax.xml.parsers.ParserConfigurationException: Feature '
>>>>>>> http://apache.org/xml/features/xinclude' is not recognized.
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>>>>>>> Source)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
>>>>>>>
>>>>>>> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>>>>>>
>>>>>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>
>>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>
>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>>>>
>>>>>>> 2018-06-16 19:25:10,620 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>>>>>> - Error while getting queue information from YARN: null
>>>>>>>
>>>>>>> 2018-06-16 19:25:10,621 DEBUG
>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor -
>>>>>>> Error details
>>>>>>>
>>>>>>> java.lang.ExceptionInInitializerError
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>
>>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Garvit Sharma
>>>>> github.com/garvitlnmiit/
>>>>>
>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>> determination that makes him master.
>>>>>
>>>>
>>>>
>>>
>>> --
>>>
>>> Garvit Sharma
>>> github.com/garvitlnmiit/
>>>
>>> No Body is a Scholar by birth, its only hard work and strong
>>> determination that makes him master.
>>>
>>
>
> --
>
> Garvit Sharma
> github.com/garvitlnmiit/
>
> No Body is a Scholar by birth, its only hard work and strong determination
> that makes him master.
>
Re: Exception while submitting jobs through Yarn
Posted by Garvit Sharma <ga...@gmail.com>.
Hi,
Sorry for the confusion, but the yarn is running on Hadoop version 2.7 only
and hence I am using Flink 1.5 Hadoop 2.7 binary.
Below are the details provided by Yarn version command :
Hadoop 2.7.3.2.6.3.0-235
Subversion git@github.com:hortonworks/hadoop.git -r
45bfd33bba8acadfa0e6024c80981c023b28d454
Compiled by jenkins on 2017-10-30T02:31Z
Compiled with protoc 2.5.0
From source with checksum cd1a4a466ef450f547c279989f3aa3
This command was run using
/usr/hdp/2.6.3.0-235/hadoop/hadoop-common-2.7.3.2.6.3.0-235.jar
Please let me know if you have found the resolution to my issue :)
Thanks,
On Mon, Jun 18, 2018 at 4:50 PM Till Rohrmann <tr...@apache.org> wrote:
> Which Hadoop version have you installed? It looks as if Flink has been
> build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class path. If
> you want to run Flink on Hadoop 2.6, then try to use the Hadoop free Flink
> binaries or the one built for Hadoop 2.6.
>
> Cheers,
> Till
>
> On Mon, Jun 18, 2018 at 10:48 AM Garvit Sharma <ga...@gmail.com>
> wrote:
>
>> Ok, I have attached the log file.
>>
>> Please check and let me know.
>>
>> Thanks,
>>
>> On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <aj...@gmail.com> wrote:
>>
>>> Hi Gravit,
>>>
>>> I think Till is interested to know about classpath details present at
>>> the start of JM and TM logs e.g. following logs provide classpath details
>>> used by TM in our case.
>>>
>>> 2018-06-17 19:01:30,656 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>> --------------------------------------------------------------------------------
>>> 2018-06-17 19:01:30,658 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Starting
>>> YARN TaskExecutor runner (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @
>>> 14:54:44 UTC)
>>> 2018-06-17 19:01:30,659 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner - OS
>>> current user: yarn
>>> 2018-06-17 19:01:31,662 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Current
>>> Hadoop/Kerberos user: hadoop
>>> 2018-06-17 19:01:31,663 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM:
>>> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10
>>> 2018-06-17 19:01:31,663 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Maximum
>>> heap size: 6647 MiBytes
>>> 2018-06-17 19:01:31,663 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>> JAVA_HOME: /usr/lib/jvm/java-openjdk
>>> 2018-06-17 19:01:31,664 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Hadoop
>>> version: 2.8.3
>>> 2018-06-17 19:01:31,664 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM
>>> Options:
>>> 2018-06-17 19:01:31,665 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>> -Xms6936m
>>> 2018-06-17 19:01:31,665 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>> -Xmx6936m
>>> 2018-06-17 19:01:31,665 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>> -XX:MaxDirectMemorySize=4072m
>>> 2018-06-17 19:01:31,665 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>> -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log
>>> 2018-06-17 19:01:31,665 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>> -Dlogback.configurationFile=file:./logback.xml
>>> 2018-06-17 19:01:31,665 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>> -Dlog4j.configuration=file:./log4j.properties
>>> 2018-06-17 19:01:31,665 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Program
>>> Arguments:
>>> 2018-06-17 19:01:31,665 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>> --configDir
>>> 2018-06-17 19:01:31,665 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner - .
>>> *2018-06-17 19:01:31,666 INFO
>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>> Classpath:
>>> lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........*
>>>
>>> --
>>> Thanks,
>>> Amit
>>>
>>> On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <ga...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Please refer to my previous mail for complete logs.
>>>>
>>>> Thanks,
>>>>
>>>> On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <tr...@apache.org>
>>>> wrote:
>>>>
>>>>> Could you also please share the complete log file with us.
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> The error for core-default.xml is interesting.
>>>>>>
>>>>>> Flink doesn't have this file. Probably it came with Yarn. Please
>>>>>> check the hadoop version Flink was built with versus the hadoop version in
>>>>>> your cluster.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> -------- Original message --------
>>>>>> From: Garvit Sharma <ga...@gmail.com>
>>>>>> Date: 6/16/18 7:23 AM (GMT-08:00)
>>>>>> To: trohrmann@apache.org
>>>>>> Cc: Chesnay Schepler <ch...@apache.org>, user@flink.apache.org
>>>>>> Subject: Re: Exception while submitting jobs through Yarn
>>>>>>
>>>>>> I am not able to figure out, got stuck badly in this since last 1
>>>>>> week. Any little help would be appreciated.
>>>>>>
>>>>>>
>>>>>> 2018-06-16 19:25:10,523 DEBUG
>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>> Parallelism set: 1 for 8
>>>>>>
>>>>>> 2018-06-16 19:25:10,578 DEBUG
>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>> Parallelism set: 1 for 1
>>>>>>
>>>>>> 2018-06-16 19:25:10,588 DEBUG
>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>> CONNECTED: KeyGroupStreamPartitioner - 1 -> 8
>>>>>>
>>>>>> 2018-06-16 19:25:10,591 DEBUG
>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>> Parallelism set: 1 for 5
>>>>>>
>>>>>> 2018-06-16 19:25:10,597 DEBUG
>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>> CONNECTED: KeyGroupStreamPartitioner - 5 -> 8
>>>>>>
>>>>>> 2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration
>>>>>> - error parsing conf core-default.xml
>>>>>>
>>>>>> javax.xml.parsers.ParserConfigurationException: Feature '
>>>>>> http://apache.org/xml/features/xinclude' is not recognized.
>>>>>>
>>>>>> at
>>>>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>>>>>> Source)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
>>>>>>
>>>>>> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>>>>>
>>>>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>
>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>
>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>>
>>>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>>>
>>>>>> 2018-06-16 19:25:10,620 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>>>>> - Error while getting queue information from YARN: null
>>>>>>
>>>>>> 2018-06-16 19:25:10,621 DEBUG
>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor -
>>>>>> Error details
>>>>>>
>>>>>> java.lang.ExceptionInInitializerError
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>
>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Garvit Sharma
>>>> github.com/garvitlnmiit/
>>>>
>>>> No Body is a Scholar by birth, its only hard work and strong
>>>> determination that makes him master.
>>>>
>>>
>>>
>>
>> --
>>
>> Garvit Sharma
>> github.com/garvitlnmiit/
>>
>> No Body is a Scholar by birth, its only hard work and strong
>> determination that makes him master.
>>
>
--
Garvit Sharma
github.com/garvitlnmiit/
No Body is a Scholar by birth, its only hard work and strong determination
that makes him master.
Re: Exception while submitting jobs through Yarn
Posted by Till Rohrmann <tr...@apache.org>.
Which Hadoop version have you installed? It looks as if Flink has been
build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class path. If
you want to run Flink on Hadoop 2.6, then try to use the Hadoop free Flink
binaries or the one built for Hadoop 2.6.
Cheers,
Till
On Mon, Jun 18, 2018 at 10:48 AM Garvit Sharma <ga...@gmail.com> wrote:
> Ok, I have attached the log file.
>
> Please check and let me know.
>
> Thanks,
>
> On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <aj...@gmail.com> wrote:
>
>> Hi Gravit,
>>
>> I think Till is interested to know about classpath details present at the
>> start of JM and TM logs e.g. following logs provide classpath details used
>> by TM in our case.
>>
>> 2018-06-17 19:01:30,656 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>> --------------------------------------------------------------------------------
>> 2018-06-17 19:01:30,658 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner - Starting
>> YARN TaskExecutor runner (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @
>> 14:54:44 UTC)
>> 2018-06-17 19:01:30,659 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner - OS
>> current user: yarn
>> 2018-06-17 19:01:31,662 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner - Current
>> Hadoop/Kerberos user: hadoop
>> 2018-06-17 19:01:31,663 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM:
>> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10
>> 2018-06-17 19:01:31,663 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner - Maximum
>> heap size: 6647 MiBytes
>> 2018-06-17 19:01:31,663 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>> JAVA_HOME: /usr/lib/jvm/java-openjdk
>> 2018-06-17 19:01:31,664 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner - Hadoop
>> version: 2.8.3
>> 2018-06-17 19:01:31,664 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM
>> Options:
>> 2018-06-17 19:01:31,665 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>> -Xms6936m
>> 2018-06-17 19:01:31,665 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>> -Xmx6936m
>> 2018-06-17 19:01:31,665 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>> -XX:MaxDirectMemorySize=4072m
>> 2018-06-17 19:01:31,665 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>> -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log
>> 2018-06-17 19:01:31,665 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>> -Dlogback.configurationFile=file:./logback.xml
>> 2018-06-17 19:01:31,665 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>> -Dlog4j.configuration=file:./log4j.properties
>> 2018-06-17 19:01:31,665 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner - Program
>> Arguments:
>> 2018-06-17 19:01:31,665 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>> --configDir
>> 2018-06-17 19:01:31,665 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner - .
>> *2018-06-17 19:01:31,666 INFO
>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>> Classpath:
>> lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........*
>>
>> --
>> Thanks,
>> Amit
>>
>> On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <ga...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Please refer to my previous mail for complete logs.
>>>
>>> Thanks,
>>>
>>> On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <tr...@apache.org>
>>> wrote:
>>>
>>>> Could you also please share the complete log file with us.
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> The error for core-default.xml is interesting.
>>>>>
>>>>> Flink doesn't have this file. Probably it came with Yarn. Please check
>>>>> the hadoop version Flink was built with versus the hadoop version in your
>>>>> cluster.
>>>>>
>>>>> Thanks
>>>>>
>>>>> -------- Original message --------
>>>>> From: Garvit Sharma <ga...@gmail.com>
>>>>> Date: 6/16/18 7:23 AM (GMT-08:00)
>>>>> To: trohrmann@apache.org
>>>>> Cc: Chesnay Schepler <ch...@apache.org>, user@flink.apache.org
>>>>> Subject: Re: Exception while submitting jobs through Yarn
>>>>>
>>>>> I am not able to figure out, got stuck badly in this since last 1
>>>>> week. Any little help would be appreciated.
>>>>>
>>>>>
>>>>> 2018-06-16 19:25:10,523 DEBUG
>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>> Parallelism set: 1 for 8
>>>>>
>>>>> 2018-06-16 19:25:10,578 DEBUG
>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>> Parallelism set: 1 for 1
>>>>>
>>>>> 2018-06-16 19:25:10,588 DEBUG
>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>> CONNECTED: KeyGroupStreamPartitioner - 1 -> 8
>>>>>
>>>>> 2018-06-16 19:25:10,591 DEBUG
>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>> Parallelism set: 1 for 5
>>>>>
>>>>> 2018-06-16 19:25:10,597 DEBUG
>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>> CONNECTED: KeyGroupStreamPartitioner - 5 -> 8
>>>>>
>>>>> 2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration
>>>>> - error parsing conf core-default.xml
>>>>>
>>>>> javax.xml.parsers.ParserConfigurationException: Feature '
>>>>> http://apache.org/xml/features/xinclude' is not recognized.
>>>>>
>>>>> at
>>>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>>>>> Source)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
>>>>>
>>>>> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>>>>
>>>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>
>>>>> at
>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>
>>>>> at
>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>
>>>>> at
>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>
>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>
>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
>>>>>
>>>>> at
>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>
>>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>>
>>>>> 2018-06-16 19:25:10,620 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>>>> - Error while getting queue information from YARN: null
>>>>>
>>>>> 2018-06-16 19:25:10,621 DEBUG
>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Error
>>>>> details
>>>>>
>>>>> java.lang.ExceptionInInitializerError
>>>>>
>>>>> at
>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>
>>>>> at
>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>
>>>>> at
>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>
>>>>> at
>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>
>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>
>>>>
>>>
>>> --
>>>
>>> Garvit Sharma
>>> github.com/garvitlnmiit/
>>>
>>> No Body is a Scholar by birth, its only hard work and strong
>>> determination that makes him master.
>>>
>>
>>
>
> --
>
> Garvit Sharma
> github.com/garvitlnmiit/
>
> No Body is a Scholar by birth, its only hard work and strong determination
> that makes him master.
>
Re: Exception while submitting jobs through Yarn
Posted by Garvit Sharma <ga...@gmail.com>.
Ok, I have attached the log file.
Please check and let me know.
Thanks,
On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <aj...@gmail.com> wrote:
> Hi Gravit,
>
> I think Till is interested to know about classpath details present at the
> start of JM and TM logs e.g. following logs provide classpath details used
> by TM in our case.
>
> 2018-06-17 19:01:30,656 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> -
> --------------------------------------------------------------------------------
> 2018-06-17 19:01:30,658 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - Starting YARN TaskExecutor runner (Version: 1.5.0,
> Rev:c61b108, Date:24.05.2018 @ 14:54:44 UTC)
> 2018-06-17 19:01:30,659 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - OS current user: yarn
> 2018-06-17 19:01:31,662 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - Current Hadoop/Kerberos user: hadoop
> 2018-06-17 19:01:31,663 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation -
> 1.8/25.171-b10
> 2018-06-17 19:01:31,663 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - Maximum heap size: 6647 MiBytes
> 2018-06-17 19:01:31,663 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - JAVA_HOME: /usr/lib/jvm/java-openjdk
> 2018-06-17 19:01:31,664 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - Hadoop version: 2.8.3
> 2018-06-17 19:01:31,664 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - JVM Options:
> 2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - -Xms6936m
> 2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - -Xmx6936m
> 2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - -XX:MaxDirectMemorySize=4072m
> 2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> -
> -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log
> 2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - -Dlogback.configurationFile=file:./logback.xml
> 2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - -Dlog4j.configuration=file:./log4j.properties
> 2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - Program Arguments:
> 2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - --configDir
> 2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
> - .
> *2018-06-17 19:01:31,666 INFO
> org.apache.flink.yarn.YarnTaskExecutorRunner -
> Classpath:
> lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........*
>
> --
> Thanks,
> Amit
>
> On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <ga...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Please refer to my previous mail for complete logs.
>>
>> Thanks,
>>
>> On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>>> Could you also please share the complete log file with us.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> The error for core-default.xml is interesting.
>>>>
>>>> Flink doesn't have this file. Probably it came with Yarn. Please check
>>>> the hadoop version Flink was built with versus the hadoop version in your
>>>> cluster.
>>>>
>>>> Thanks
>>>>
>>>> -------- Original message --------
>>>> From: Garvit Sharma <ga...@gmail.com>
>>>> Date: 6/16/18 7:23 AM (GMT-08:00)
>>>> To: trohrmann@apache.org
>>>> Cc: Chesnay Schepler <ch...@apache.org>, user@flink.apache.org
>>>> Subject: Re: Exception while submitting jobs through Yarn
>>>>
>>>> I am not able to figure out, got stuck badly in this since last 1 week.
>>>> Any little help would be appreciated.
>>>>
>>>>
>>>> 2018-06-16 19:25:10,523 DEBUG
>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>> Parallelism set: 1 for 8
>>>>
>>>> 2018-06-16 19:25:10,578 DEBUG
>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>> Parallelism set: 1 for 1
>>>>
>>>> 2018-06-16 19:25:10,588 DEBUG
>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>> CONNECTED: KeyGroupStreamPartitioner - 1 -> 8
>>>>
>>>> 2018-06-16 19:25:10,591 DEBUG
>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>> Parallelism set: 1 for 5
>>>>
>>>> 2018-06-16 19:25:10,597 DEBUG
>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>> CONNECTED: KeyGroupStreamPartitioner - 5 -> 8
>>>>
>>>> 2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration
>>>> - error parsing conf core-default.xml
>>>>
>>>> javax.xml.parsers.ParserConfigurationException: Feature '
>>>> http://apache.org/xml/features/xinclude' is not recognized.
>>>>
>>>> at
>>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>>>> Source)
>>>>
>>>> at
>>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
>>>>
>>>> at
>>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
>>>>
>>>> at
>>>> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
>>>>
>>>> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
>>>>
>>>> at
>>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>>>
>>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>>>
>>>> at
>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>
>>>> at
>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>
>>>> at
>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>
>>>> at
>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>
>>>> at
>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>
>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>
>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
>>>>
>>>> at
>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>
>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>
>>>> 2018-06-16 19:25:10,620 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>>> - Error while getting queue information from YARN: null
>>>>
>>>> 2018-06-16 19:25:10,621 DEBUG
>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Error
>>>> details
>>>>
>>>> java.lang.ExceptionInInitializerError
>>>>
>>>> at
>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>
>>>> at
>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>
>>>> at
>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>
>>>> at
>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>
>>>> at
>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>
>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>
>>>
>>
>> --
>>
>> Garvit Sharma
>> github.com/garvitlnmiit/
>>
>> No Body is a Scholar by birth, its only hard work and strong
>> determination that makes him master.
>>
>
>
--
Garvit Sharma
github.com/garvitlnmiit/
No Body is a Scholar by birth, its only hard work and strong determination
that makes him master.
Re: Exception while submitting jobs through Yarn
Posted by Amit Jain <aj...@gmail.com>.
Hi Gravit,
I think Till is interested to know about classpath details present at the
start of JM and TM logs e.g. following logs provide classpath details used
by TM in our case.
2018-06-17 19:01:30,656 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
-
--------------------------------------------------------------------------------
2018-06-17 19:01:30,658 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Starting YARN TaskExecutor runner (Version: 1.5.0,
Rev:c61b108, Date:24.05.2018 @ 14:54:44 UTC)
2018-06-17 19:01:30,659 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- OS current user: yarn
2018-06-17 19:01:31,662 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Current Hadoop/Kerberos user: hadoop
2018-06-17 19:01:31,663 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- JVM: OpenJDK 64-Bit Server VM - Oracle Corporation -
1.8/25.171-b10
2018-06-17 19:01:31,663 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Maximum heap size: 6647 MiBytes
2018-06-17 19:01:31,663 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- JAVA_HOME: /usr/lib/jvm/java-openjdk
2018-06-17 19:01:31,664 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Hadoop version: 2.8.3
2018-06-17 19:01:31,664 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- JVM Options:
2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Xms6936m
2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Xmx6936m
2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -XX:MaxDirectMemorySize=4072m
2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
-
-Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log
2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Dlogback.configurationFile=file:./logback.xml
2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- -Dlog4j.configuration=file:./log4j.properties
2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Program Arguments:
2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- --configDir
2018-06-17 19:01:31,665 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- .
*2018-06-17 19:01:31,666 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Classpath:
lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........*
--
Thanks,
Amit
On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <ga...@gmail.com> wrote:
> Hi,
>
> Please refer to my previous mail for complete logs.
>
> Thanks,
>
> On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Could you also please share the complete log file with us.
>>
>> Cheers,
>> Till
>>
>> On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yu...@gmail.com> wrote:
>>
>>> The error for core-default.xml is interesting.
>>>
>>> Flink doesn't have this file. Probably it came with Yarn. Please check
>>> the hadoop version Flink was built with versus the hadoop version in your
>>> cluster.
>>>
>>> Thanks
>>>
>>> -------- Original message --------
>>> From: Garvit Sharma <ga...@gmail.com>
>>> Date: 6/16/18 7:23 AM (GMT-08:00)
>>> To: trohrmann@apache.org
>>> Cc: Chesnay Schepler <ch...@apache.org>, user@flink.apache.org
>>> Subject: Re: Exception while submitting jobs through Yarn
>>>
>>> I am not able to figure out, got stuck badly in this since last 1 week.
>>> Any little help would be appreciated.
>>>
>>>
>>> 2018-06-16 19:25:10,523 DEBUG org.apache.flink.streaming.api.graph.
>>> StreamingJobGraphGenerator - Parallelism set: 1 for 8
>>>
>>> 2018-06-16 19:25:10,578 DEBUG org.apache.flink.streaming.api.graph.
>>> StreamingJobGraphGenerator - Parallelism set: 1 for 1
>>>
>>> 2018-06-16 19:25:10,588 DEBUG org.apache.flink.streaming.api.graph.
>>> StreamingJobGraphGenerator - CONNECTED: KeyGroupStreamPartitioner - 1
>>> -> 8
>>>
>>> 2018-06-16 19:25:10,591 DEBUG org.apache.flink.streaming.api.graph.
>>> StreamingJobGraphGenerator - Parallelism set: 1 for 5
>>>
>>> 2018-06-16 19:25:10,597 DEBUG org.apache.flink.streaming.api.graph.
>>> StreamingJobGraphGenerator - CONNECTED: KeyGroupStreamPartitioner - 5
>>> -> 8
>>>
>>> 2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration
>>> - error parsing conf core-default.xml
>>>
>>> javax.xml.parsers.ParserConfigurationException: Feature '
>>> http://apache.org/xml/features/xinclude' is not recognized.
>>>
>>> at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>>> Source)
>>>
>>> at org.apache.hadoop.conf.Configuration.loadResource(
>>> Configuration.java:2482)
>>>
>>> at org.apache.hadoop.conf.Configuration.loadResources(
>>> Configuration.java:2444)
>>>
>>> at org.apache.hadoop.conf.Configuration.getProps(
>>> Configuration.java:2361)
>>>
>>> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
>>>
>>> at org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.
>>> getRecordFactory(RecordFactoryProvider.java:49)
>>>
>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>>
>>> at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.
>>> getQueueInfoRequest(YarnClientImpl.java:495)
>>>
>>> at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.
>>> getAllQueues(YarnClientImpl.java:525)
>>>
>>> at org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(
>>> AbstractYarnClusterDescriptor.java:658)
>>>
>>> at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(
>>> AbstractYarnClusterDescriptor.java:486)
>>>
>>> at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(
>>> YarnClusterDescriptor.java:75)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.runProgram(
>>> CliFrontend.java:235)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.parseParameters(
>>> CliFrontend.java:1020)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.lambda$main$9(
>>> CliFrontend.java:1096)
>>>
>>> at java.security.AccessController.doPrivileged(Native Method)
>>>
>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>
>>> at org.apache.hadoop.security.UserGroupInformation.doAs(
>>> UserGroupInformation.java:1692)
>>>
>>> at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(
>>> HadoopSecurityContext.java:41)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>
>>> 2018-06-16 19:25:10,620 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>> - Error while getting queue information from YARN: null
>>>
>>> 2018-06-16 19:25:10,621 DEBUG org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>> - Error details
>>>
>>> java.lang.ExceptionInInitializerError
>>>
>>> at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.
>>> getQueueInfoRequest(YarnClientImpl.java:495)
>>>
>>> at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.
>>> getAllQueues(YarnClientImpl.java:525)
>>>
>>> at org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(
>>> AbstractYarnClusterDescriptor.java:658)
>>>
>>> at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(
>>> AbstractYarnClusterDescriptor.java:486)
>>>
>>> at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(
>>> YarnClusterDescriptor.java:75)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.runProgram(
>>> CliFrontend.java:235)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.parseParameters(
>>> CliFrontend.java:1020)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.lambda$main$9(
>>> CliFrontend.java:1096)
>>>
>>
>
> --
>
> Garvit Sharma
> github.com/garvitlnmiit/
>
> No Body is a Scholar by birth, its only hard work and strong determination
> that makes him master.
>
Re: Exception while submitting jobs through Yarn
Posted by Garvit Sharma <ga...@gmail.com>.
Hi,
Please refer to my previous mail for complete logs.
Thanks,
On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <tr...@apache.org> wrote:
> Could you also please share the complete log file with us.
>
> Cheers,
> Till
>
> On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yu...@gmail.com> wrote:
>
>> The error for core-default.xml is interesting.
>>
>> Flink doesn't have this file. Probably it came with Yarn. Please check
>> the hadoop version Flink was built with versus the hadoop version in your
>> cluster.
>>
>> Thanks
>>
>> -------- Original message --------
>> From: Garvit Sharma <ga...@gmail.com>
>> Date: 6/16/18 7:23 AM (GMT-08:00)
>> To: trohrmann@apache.org
>> Cc: Chesnay Schepler <ch...@apache.org>, user@flink.apache.org
>> Subject: Re: Exception while submitting jobs through Yarn
>>
>> I am not able to figure out, got stuck badly in this since last 1 week.
>> Any little help would be appreciated.
>>
>>
>> 2018-06-16 19:25:10,523 DEBUG
>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>> Parallelism set: 1 for 8
>>
>> 2018-06-16 19:25:10,578 DEBUG
>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>> Parallelism set: 1 for 1
>>
>> 2018-06-16 19:25:10,588 DEBUG
>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>> CONNECTED: KeyGroupStreamPartitioner - 1 -> 8
>>
>> 2018-06-16 19:25:10,591 DEBUG
>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>> Parallelism set: 1 for 5
>>
>> 2018-06-16 19:25:10,597 DEBUG
>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>> CONNECTED: KeyGroupStreamPartitioner - 5 -> 8
>>
>> 2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration
>> - error parsing conf core-default.xml
>>
>> javax.xml.parsers.ParserConfigurationException: Feature '
>> http://apache.org/xml/features/xinclude' is not recognized.
>>
>> at
>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>> Source)
>>
>> at
>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
>>
>> at
>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
>>
>> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
>>
>> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
>>
>> at
>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>
>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>
>> at
>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>
>> at
>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>
>> at
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>
>> at
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>
>> at
>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>
>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
>>
>> at
>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>
>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>
>> 2018-06-16 19:25:10,620 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>> - Error while getting queue information from YARN: null
>>
>> 2018-06-16 19:25:10,621 DEBUG
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Error
>> details
>>
>> java.lang.ExceptionInInitializerError
>>
>> at
>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>
>> at
>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>
>> at
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>
>> at
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>
>> at
>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>
>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>
>
--
Garvit Sharma
github.com/garvitlnmiit/
No Body is a Scholar by birth, its only hard work and strong determination
that makes him master.
Re: Exception while submitting jobs through Yarn
Posted by Till Rohrmann <tr...@apache.org>.
Could you also please share the complete log file with us.
Cheers,
Till
On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yu...@gmail.com> wrote:
> The error for core-default.xml is interesting.
>
> Flink doesn't have this file. Probably it came with Yarn. Please check the
> hadoop version Flink was built with versus the hadoop version in your
> cluster.
>
> Thanks
>
> -------- Original message --------
> From: Garvit Sharma <ga...@gmail.com>
> Date: 6/16/18 7:23 AM (GMT-08:00)
> To: trohrmann@apache.org
> Cc: Chesnay Schepler <ch...@apache.org>, user@flink.apache.org
> Subject: Re: Exception while submitting jobs through Yarn
>
> I am not able to figure out, got stuck badly in this since last 1 week.
> Any little help would be appreciated.
>
>
> 2018-06-16 19:25:10,523 DEBUG
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
> Parallelism set: 1 for 8
>
> 2018-06-16 19:25:10,578 DEBUG
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
> Parallelism set: 1 for 1
>
> 2018-06-16 19:25:10,588 DEBUG
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
> CONNECTED: KeyGroupStreamPartitioner - 1 -> 8
>
> 2018-06-16 19:25:10,591 DEBUG
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
> Parallelism set: 1 for 5
>
> 2018-06-16 19:25:10,597 DEBUG
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
> CONNECTED: KeyGroupStreamPartitioner - 5 -> 8
>
> 2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration
> - error parsing conf core-default.xml
>
> javax.xml.parsers.ParserConfigurationException: Feature '
> http://apache.org/xml/features/xinclude' is not recognized.
>
> at
> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
> Source)
>
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
>
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
>
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
>
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
>
> at
> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>
> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>
> at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>
> at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>
> at
> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>
> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
>
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>
> 2018-06-16 19:25:10,620 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
> - Error while getting queue information from YARN: null
>
> 2018-06-16 19:25:10,621 DEBUG
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Error
> details
>
> java.lang.ExceptionInInitializerError
>
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>
> at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>
> at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>
> at
> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>
> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>
Re: Exception while submitting jobs through Yarn
Posted by Ted Yu <yu...@gmail.com>.
The error for core-default.xml is interesting.
Flink doesn't have this file. Probably it came with Yarn. Please check the hadoop version Flink was built with versus the hadoop version in your cluster.
Thanks
-------- Original message --------From: Garvit Sharma <ga...@gmail.com> Date: 6/16/18 7:23 AM (GMT-08:00) To: trohrmann@apache.org Cc: Chesnay Schepler <ch...@apache.org>, user@flink.apache.org Subject: Re: Exception while submitting jobs through Yarn
I am not able to figure out, got stuck badly in this since last 1 week. Any little help would be appreciated.
2018-06-16 19:25:10,523 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator - Parallelism set: 1 for 8
2018-06-16 19:25:10,578 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator - Parallelism set: 1 for 1
2018-06-16 19:25:10,588 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator - CONNECTED: KeyGroupStreamPartitioner - 1 -> 8
2018-06-16 19:25:10,591 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator - Parallelism set: 1 for 5
2018-06-16 19:25:10,597 DEBUG org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator - CONNECTED: KeyGroupStreamPartitioner - 5 -> 8
2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration - error parsing conf core-default.xml
javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
at org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
at org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
2018-06-16 19:25:10,620 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor - Error while getting queue information from YARN: null
2018-06-16 19:25:10,621 DEBUG org.apache.flink.yarn.AbstractYarnClusterDescriptor - Error details
java.lang.ExceptionInInitializerError
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
at org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
at org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
Re: Exception while submitting jobs through Yarn
Posted by Garvit Sharma <ga...@gmail.com>.
I am not able to figure out, got stuck badly in this since last 1 week. Any
little help would be appreciated.
2018-06-16 19:25:10,523 DEBUG
org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
Parallelism set: 1 for 8
2018-06-16 19:25:10,578 DEBUG
org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
Parallelism set: 1 for 1
2018-06-16 19:25:10,588 DEBUG
org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
CONNECTED: KeyGroupStreamPartitioner - 1 -> 8
2018-06-16 19:25:10,591 DEBUG
org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
Parallelism set: 1 for 5
2018-06-16 19:25:10,597 DEBUG
org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
CONNECTED: KeyGroupStreamPartitioner - 5 -> 8
2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration
- error parsing conf core-default.xml
javax.xml.parsers.ParserConfigurationException: Feature '
http://apache.org/xml/features/xinclude' is not recognized.
at
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
Source)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
at
org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
at
org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
2018-06-16 19:25:10,620 WARN
org.apache.flink.yarn.AbstractYarnClusterDescriptor
- Error while getting queue information from YARN: null
2018-06-16 19:25:10,621 DEBUG
org.apache.flink.yarn.AbstractYarnClusterDescriptor - Error
details
java.lang.ExceptionInInitializerError
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
at
org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
Caused by: java.lang.RuntimeException:
javax.xml.parsers.ParserConfigurationException: Feature '
http://apache.org/xml/features/xinclude' is not recognized.
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2600)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
at
org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
... 14 more
Caused by: javax.xml.parsers.ParserConfigurationException: Feature '
http://apache.org/xml/features/xinclude' is not recognized.
at
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
Source)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
... 19 more
2018-06-16 19:25:10,627 DEBUG org.apache.hadoop.service.AbstractService
- Service:
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state STOPPED
2018-06-16 19:25:10,628 DEBUG org.apache.hadoop.ipc.Client
- stopping client from cache:
org.apache.hadoop.ipc.Client@32c726ee
2018-06-16 19:25:10,628 DEBUG org.apache.hadoop.ipc.Client
- removing client from cache:
org.apache.hadoop.ipc.Client@32c726ee
2018-06-16 19:25:10,628 DEBUG org.apache.hadoop.ipc.Client
- stopping actual client because no more references remain:
org.apache.hadoop.ipc.Client@32c726ee
2018-06-16 19:25:10,628 DEBUG org.apache.hadoop.ipc.Client
- Stopping client
2018-06-16 19:25:10,629 DEBUG org.apache.hadoop.service.AbstractService
- Service:
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl entered state
STOPPED
2018-06-16 19:25:10,630 ERROR org.apache.flink.client.cli.CliFrontend
- Fatal error while running command line interface.
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.hadoop.yarn.util.Records
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:212)
at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:497)
at
org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
On Fri, Jun 15, 2018 at 9:35 PM Till Rohrmann <tr...@apache.org> wrote:
> Hmm could you maybe share the client logs with us.
>
> Cheers,
> Till
>
> On Fri, Jun 15, 2018 at 4:54 PM Garvit Sharma <ga...@gmail.com> wrote:
>
>> Yes, I did.
>>
>> On Fri, Jun 15, 2018 at 6:17 PM Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>>> Hi Garvit,
>>>
>>> have you exported the HADOOP_CLASSPATH as described in the release notes
>>> [1]?
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.5/release-notes/flink-1.5.html#hadoop-classpath-discovery
>>>
>>> Cheers,
>>> Till
>>>
>>> On Fri, Jun 15, 2018 at 2:22 PM Garvit Sharma <ga...@gmail.com>
>>> wrote:
>>>
>>>> Does someone has any idea how to get rid if the above parse exception
>>>> while submitting flink job to Yarn.
>>>>
>>>> Already searched on the internet, could not find any solution to it.
>>>>
>>>> Please help.
>>>>
>>>> On Fri, Jun 15, 2018 at 9:15 AM Garvit Sharma <ga...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks Chesnay, Now it is connecting to the Resource Manager but I am
>>>>> getting the below exception :
>>>>>
>>>>> 2018-06-15 09:12:44,812 FATAL org.apache.hadoop.conf.Configuration
>>>>> - error parsing conf core-default.xml
>>>>>
>>>>> javax.xml.parsers.ParserConfigurationException: Feature '
>>>>> http://apache.org/xml/features/xinclude' is not recognized.
>>>>>
>>>>> at
>>>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>>>>> Source)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
>>>>>
>>>>> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1350)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>>>>
>>>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:584)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:614)
>>>>>
>>>>> at
>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>
>>>>> at
>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>
>>>>> at
>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>
>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>
>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>>>>>
>>>>> at
>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>
>>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>>
>>>>> 2018-06-15 09:12:44,825 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>>>> - Error while getting queue information from YARN: null
>>>>>
>>>>> java.lang.NoClassDefFoundError: Could not initialize class
>>>>> org.apache.hadoop.yarn.util.Records
>>>>>
>>>>> at
>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:230)
>>>>>
>>>>> at
>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:497)
>>>>>
>>>>> at
>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>
>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>
>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>>>>>
>>>>> at
>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>
>>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>>
>>>>> Please help.
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>> On Thu, Jun 14, 2018 at 1:28 PM Chesnay Schepler <ch...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> My gut feeling is that these classes must be present in jars in the
>>>>>> /lib directory. I don't think you can supply these with the submitted jar.
>>>>>> For a simple test, put your jar into the /lib folder before
>>>>>> submitting it.
>>>>>>
>>>>>> On 14.06.2018 06:56, Garvit Sharma wrote:
>>>>>>
>>>>>> Can someone please tell why am I facing this?
>>>>>>
>>>>>> On Wed, Jun 13, 2018 at 10:33 PM Garvit Sharma <ga...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am using *flink-1.5.0-bin-hadoop27-scala_2.11 *to submit jobs
>>>>>>> through Yarn, but I am getting the below exception :
>>>>>>>
>>>>>>> java.lang.NoClassDefFoundError:
>>>>>>> com/sun/jersey/core/util/FeaturesAndProperties
>>>>>>>
>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>>>
>>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>>>>>>
>>>>>>> at
>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>>>>
>>>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>>>>>>
>>>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>>>>>>
>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>>>>>>
>>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>>>>>>
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>
>>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>>>>>>
>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>
>>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>>>>>>
>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.getClusterDescriptor(FlinkYarnSessionCli.java:966)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createDescriptor(FlinkYarnSessionCli.java:269)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:444)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:92)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:221)
>>>>>>>
>>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>
>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>>>>
>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>> com.sun.jersey.core.util.FeaturesAndProperties
>>>>>>>
>>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>>>>>>
>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>
>>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>>>>>>
>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>
>>>>>>>
>>>>>>> Command : HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -m
>>>>>>> yarn-cluster -yd -yn 2 -ys 20 -yjm 10240 -ytm 10240 -yst -ynm test -yqu
>>>>>>> default -p 20 test.jar
>>>>>>>
>>>>>>> The class *com/sun/jersey/core/util/FeaturesAndProperties* is
>>>>>>> already present in the test.jar so not sure why am I getting this exception.
>>>>>>>
>>>>>>> Please check and let me know.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> --
>>>>>>>
>>>>>>> Garvit Sharma
>>>>>>> github.com/garvitlnmiit/
>>>>>>>
>>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>>> determination that makes him master.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Garvit Sharma
>>>>>> github.com/garvitlnmiit/
>>>>>>
>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>> determination that makes him master.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Garvit Sharma
>>>>> github.com/garvitlnmiit/
>>>>>
>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>> determination that makes him master.
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Garvit Sharma
>>>> github.com/garvitlnmiit/
>>>>
>>>> No Body is a Scholar by birth, its only hard work and strong
>>>> determination that makes him master.
>>>>
>>>
>>
>> --
>>
>> Garvit Sharma
>> github.com/garvitlnmiit/
>>
>> No Body is a Scholar by birth, its only hard work and strong
>> determination that makes him master.
>>
>
--
Garvit Sharma
github.com/garvitlnmiit/
No Body is a Scholar by birth, its only hard work and strong determination
that makes him master.
Re: Exception while submitting jobs through Yarn
Posted by Till Rohrmann <tr...@apache.org>.
Hmm could you maybe share the client logs with us.
Cheers,
Till
On Fri, Jun 15, 2018 at 4:54 PM Garvit Sharma <ga...@gmail.com> wrote:
> Yes, I did.
>
> On Fri, Jun 15, 2018 at 6:17 PM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Hi Garvit,
>>
>> have you exported the HADOOP_CLASSPATH as described in the release notes
>> [1]?
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-release-1.5/release-notes/flink-1.5.html#hadoop-classpath-discovery
>>
>> Cheers,
>> Till
>>
>> On Fri, Jun 15, 2018 at 2:22 PM Garvit Sharma <ga...@gmail.com>
>> wrote:
>>
>>> Does someone has any idea how to get rid if the above parse exception
>>> while submitting flink job to Yarn.
>>>
>>> Already searched on the internet, could not find any solution to it.
>>>
>>> Please help.
>>>
>>> On Fri, Jun 15, 2018 at 9:15 AM Garvit Sharma <ga...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Chesnay, Now it is connecting to the Resource Manager but I am
>>>> getting the below exception :
>>>>
>>>> 2018-06-15 09:12:44,812 FATAL org.apache.hadoop.conf.Configuration
>>>> - error parsing conf core-default.xml
>>>>
>>>> javax.xml.parsers.ParserConfigurationException: Feature '
>>>> http://apache.org/xml/features/xinclude' is not recognized.
>>>>
>>>> at
>>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>>>> Source)
>>>>
>>>> at
>>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737)
>>>>
>>>> at
>>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696)
>>>>
>>>> at
>>>> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
>>>>
>>>> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1350)
>>>>
>>>> at
>>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>>>
>>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>>>
>>>> at
>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:584)
>>>>
>>>> at
>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:614)
>>>>
>>>> at
>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>
>>>> at
>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>
>>>> at
>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>
>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>
>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>>>>
>>>> at
>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>
>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>
>>>> 2018-06-15 09:12:44,825 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>>> - Error while getting queue information from YARN: null
>>>>
>>>> java.lang.NoClassDefFoundError: Could not initialize class
>>>> org.apache.hadoop.yarn.util.Records
>>>>
>>>> at
>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:230)
>>>>
>>>> at
>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:497)
>>>>
>>>> at
>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>
>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>
>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>>>>
>>>> at
>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>
>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>
>>>> Please help.
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> On Thu, Jun 14, 2018 at 1:28 PM Chesnay Schepler <ch...@apache.org>
>>>> wrote:
>>>>
>>>>> My gut feeling is that these classes must be present in jars in the
>>>>> /lib directory. I don't think you can supply these with the submitted jar.
>>>>> For a simple test, put your jar into the /lib folder before submitting
>>>>> it.
>>>>>
>>>>> On 14.06.2018 06:56, Garvit Sharma wrote:
>>>>>
>>>>> Can someone please tell why am I facing this?
>>>>>
>>>>> On Wed, Jun 13, 2018 at 10:33 PM Garvit Sharma <ga...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am using *flink-1.5.0-bin-hadoop27-scala_2.11 *to submit jobs
>>>>>> through Yarn, but I am getting the below exception :
>>>>>>
>>>>>> java.lang.NoClassDefFoundError:
>>>>>> com/sun/jersey/core/util/FeaturesAndProperties
>>>>>>
>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>>
>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>>>>>
>>>>>> at
>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>>>
>>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>>>>>
>>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>>>>>
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>>>>>
>>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>>>>>
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>>>>>
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>>>>>
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.getClusterDescriptor(FlinkYarnSessionCli.java:966)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createDescriptor(FlinkYarnSessionCli.java:269)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:444)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:92)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:221)
>>>>>>
>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>
>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>>
>>>>>> at
>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
>>>>>>
>>>>>> at
>>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>>
>>>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>>>
>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>> com.sun.jersey.core.util.FeaturesAndProperties
>>>>>>
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>>>>>
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>>>>>
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>
>>>>>>
>>>>>> Command : HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -m
>>>>>> yarn-cluster -yd -yn 2 -ys 20 -yjm 10240 -ytm 10240 -yst -ynm test -yqu
>>>>>> default -p 20 test.jar
>>>>>>
>>>>>> The class *com/sun/jersey/core/util/FeaturesAndProperties* is
>>>>>> already present in the test.jar so not sure why am I getting this exception.
>>>>>>
>>>>>> Please check and let me know.
>>>>>>
>>>>>> Thanks,
>>>>>> --
>>>>>>
>>>>>> Garvit Sharma
>>>>>> github.com/garvitlnmiit/
>>>>>>
>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>> determination that makes him master.
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Garvit Sharma
>>>>> github.com/garvitlnmiit/
>>>>>
>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>> determination that makes him master.
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Garvit Sharma
>>>> github.com/garvitlnmiit/
>>>>
>>>> No Body is a Scholar by birth, its only hard work and strong
>>>> determination that makes him master.
>>>>
>>>
>>>
>>> --
>>>
>>> Garvit Sharma
>>> github.com/garvitlnmiit/
>>>
>>> No Body is a Scholar by birth, its only hard work and strong
>>> determination that makes him master.
>>>
>>
>
> --
>
> Garvit Sharma
> github.com/garvitlnmiit/
>
> No Body is a Scholar by birth, its only hard work and strong determination
> that makes him master.
>
Re: Exception while submitting jobs through Yarn
Posted by Garvit Sharma <ga...@gmail.com>.
Yes, I did.
On Fri, Jun 15, 2018 at 6:17 PM Till Rohrmann <tr...@apache.org> wrote:
> Hi Garvit,
>
> have you exported the HADOOP_CLASSPATH as described in the release notes
> [1]?
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.5/release-notes/flink-1.5.html#hadoop-classpath-discovery
>
> Cheers,
> Till
>
> On Fri, Jun 15, 2018 at 2:22 PM Garvit Sharma <ga...@gmail.com> wrote:
>
>> Does someone has any idea how to get rid if the above parse exception
>> while submitting flink job to Yarn.
>>
>> Already searched on the internet, could not find any solution to it.
>>
>> Please help.
>>
>> On Fri, Jun 15, 2018 at 9:15 AM Garvit Sharma <ga...@gmail.com>
>> wrote:
>>
>>> Thanks Chesnay, Now it is connecting to the Resource Manager but I am
>>> getting the below exception :
>>>
>>> 2018-06-15 09:12:44,812 FATAL org.apache.hadoop.conf.Configuration
>>> - error parsing conf core-default.xml
>>>
>>> javax.xml.parsers.ParserConfigurationException: Feature '
>>> http://apache.org/xml/features/xinclude' is not recognized.
>>>
>>> at
>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>>> Source)
>>>
>>> at
>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737)
>>>
>>> at
>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696)
>>>
>>> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
>>>
>>> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1350)
>>>
>>> at
>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>>
>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>>
>>> at
>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:584)
>>>
>>> at
>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:614)
>>>
>>> at
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>
>>> at
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>
>>> at
>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>
>>> at
>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>
>>> at
>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>
>>> at
>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>
>>> at java.security.AccessController.doPrivileged(Native Method)
>>>
>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>>>
>>> at
>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>
>>> 2018-06-15 09:12:44,825 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>> - Error while getting queue information from YARN: null
>>>
>>> java.lang.NoClassDefFoundError: Could not initialize class
>>> org.apache.hadoop.yarn.util.Records
>>>
>>> at
>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:230)
>>>
>>> at
>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:497)
>>>
>>> at
>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>
>>> at
>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>
>>> at
>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>
>>> at
>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>
>>> at java.security.AccessController.doPrivileged(Native Method)
>>>
>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>>>
>>> at
>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>
>>> Please help.
>>>
>>> Thanks,
>>>
>>>
>>> On Thu, Jun 14, 2018 at 1:28 PM Chesnay Schepler <ch...@apache.org>
>>> wrote:
>>>
>>>> My gut feeling is that these classes must be present in jars in the
>>>> /lib directory. I don't think you can supply these with the submitted jar.
>>>> For a simple test, put your jar into the /lib folder before submitting
>>>> it.
>>>>
>>>> On 14.06.2018 06:56, Garvit Sharma wrote:
>>>>
>>>> Can someone please tell why am I facing this?
>>>>
>>>> On Wed, Jun 13, 2018 at 10:33 PM Garvit Sharma <ga...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am using *flink-1.5.0-bin-hadoop27-scala_2.11 *to submit jobs
>>>>> through Yarn, but I am getting the below exception :
>>>>>
>>>>> java.lang.NoClassDefFoundError:
>>>>> com/sun/jersey/core/util/FeaturesAndProperties
>>>>>
>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>
>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>>>>
>>>>> at
>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>>
>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>>>>
>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>>>>
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>>>>
>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>>>>
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>>>>
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>>>>
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>>>>>
>>>>> at
>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.getClusterDescriptor(FlinkYarnSessionCli.java:966)
>>>>>
>>>>> at
>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createDescriptor(FlinkYarnSessionCli.java:269)
>>>>>
>>>>> at
>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:444)
>>>>>
>>>>> at
>>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:92)
>>>>>
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:221)
>>>>>
>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>
>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>
>>>>> at
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
>>>>>
>>>>> at
>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>
>>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>>
>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>> com.sun.jersey.core.util.FeaturesAndProperties
>>>>>
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>>>>
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>>>>
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>
>>>>>
>>>>> Command : HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -m
>>>>> yarn-cluster -yd -yn 2 -ys 20 -yjm 10240 -ytm 10240 -yst -ynm test -yqu
>>>>> default -p 20 test.jar
>>>>>
>>>>> The class *com/sun/jersey/core/util/FeaturesAndProperties* is already
>>>>> present in the test.jar so not sure why am I getting this exception.
>>>>>
>>>>> Please check and let me know.
>>>>>
>>>>> Thanks,
>>>>> --
>>>>>
>>>>> Garvit Sharma
>>>>> github.com/garvitlnmiit/
>>>>>
>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>> determination that makes him master.
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Garvit Sharma
>>>> github.com/garvitlnmiit/
>>>>
>>>> No Body is a Scholar by birth, its only hard work and strong
>>>> determination that makes him master.
>>>>
>>>>
>>>>
>>>
>>> --
>>>
>>> Garvit Sharma
>>> github.com/garvitlnmiit/
>>>
>>> No Body is a Scholar by birth, its only hard work and strong
>>> determination that makes him master.
>>>
>>
>>
>> --
>>
>> Garvit Sharma
>> github.com/garvitlnmiit/
>>
>> No Body is a Scholar by birth, its only hard work and strong
>> determination that makes him master.
>>
>
--
Garvit Sharma
github.com/garvitlnmiit/
No Body is a Scholar by birth, its only hard work and strong determination
that makes him master.
Re: Exception while submitting jobs through Yarn
Posted by Till Rohrmann <tr...@apache.org>.
Hi Garvit,
have you exported the HADOOP_CLASSPATH as described in the release notes
[1]?
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.5/release-notes/flink-1.5.html#hadoop-classpath-discovery
Cheers,
Till
On Fri, Jun 15, 2018 at 2:22 PM Garvit Sharma <ga...@gmail.com> wrote:
> Does someone has any idea how to get rid if the above parse exception
> while submitting flink job to Yarn.
>
> Already searched on the internet, could not find any solution to it.
>
> Please help.
>
> On Fri, Jun 15, 2018 at 9:15 AM Garvit Sharma <ga...@gmail.com> wrote:
>
>> Thanks Chesnay, Now it is connecting to the Resource Manager but I am
>> getting the below exception :
>>
>> 2018-06-15 09:12:44,812 FATAL org.apache.hadoop.conf.Configuration
>> - error parsing conf core-default.xml
>>
>> javax.xml.parsers.ParserConfigurationException: Feature '
>> http://apache.org/xml/features/xinclude' is not recognized.
>>
>> at
>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>> Source)
>>
>> at
>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737)
>>
>> at
>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696)
>>
>> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
>>
>> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1350)
>>
>> at
>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>
>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>
>> at
>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:584)
>>
>> at
>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:614)
>>
>> at
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>
>> at
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>
>> at
>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>
>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>>
>> at
>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>
>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>
>> 2018-06-15 09:12:44,825 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
>> - Error while getting queue information from YARN: null
>>
>> java.lang.NoClassDefFoundError: Could not initialize class
>> org.apache.hadoop.yarn.util.Records
>>
>> at
>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:230)
>>
>> at
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:497)
>>
>> at
>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>
>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>>
>> at
>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>
>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>
>> Please help.
>>
>> Thanks,
>>
>>
>> On Thu, Jun 14, 2018 at 1:28 PM Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>>> My gut feeling is that these classes must be present in jars in the /lib
>>> directory. I don't think you can supply these with the submitted jar.
>>> For a simple test, put your jar into the /lib folder before submitting
>>> it.
>>>
>>> On 14.06.2018 06:56, Garvit Sharma wrote:
>>>
>>> Can someone please tell why am I facing this?
>>>
>>> On Wed, Jun 13, 2018 at 10:33 PM Garvit Sharma <ga...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am using *flink-1.5.0-bin-hadoop27-scala_2.11 *to submit jobs
>>>> through Yarn, but I am getting the below exception :
>>>>
>>>> java.lang.NoClassDefFoundError:
>>>> com/sun/jersey/core/util/FeaturesAndProperties
>>>>
>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>
>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>>>
>>>> at
>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>>
>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>>>
>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>>>
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>>>
>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>>>
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>>>
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>>>
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>
>>>> at
>>>> org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
>>>>
>>>> at
>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
>>>>
>>>> at
>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
>>>>
>>>> at
>>>> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>>>>
>>>> at
>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.getClusterDescriptor(FlinkYarnSessionCli.java:966)
>>>>
>>>> at
>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createDescriptor(FlinkYarnSessionCli.java:269)
>>>>
>>>> at
>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:444)
>>>>
>>>> at
>>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:92)
>>>>
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:221)
>>>>
>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>
>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
>>>>
>>>> at
>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>
>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>
>>>> Caused by: java.lang.ClassNotFoundException:
>>>> com.sun.jersey.core.util.FeaturesAndProperties
>>>>
>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>>>
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>
>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>>>
>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>
>>>>
>>>> Command : HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -m
>>>> yarn-cluster -yd -yn 2 -ys 20 -yjm 10240 -ytm 10240 -yst -ynm test -yqu
>>>> default -p 20 test.jar
>>>>
>>>> The class *com/sun/jersey/core/util/FeaturesAndProperties* is already
>>>> present in the test.jar so not sure why am I getting this exception.
>>>>
>>>> Please check and let me know.
>>>>
>>>> Thanks,
>>>> --
>>>>
>>>> Garvit Sharma
>>>> github.com/garvitlnmiit/
>>>>
>>>> No Body is a Scholar by birth, its only hard work and strong
>>>> determination that makes him master.
>>>>
>>>
>>>
>>> --
>>>
>>> Garvit Sharma
>>> github.com/garvitlnmiit/
>>>
>>> No Body is a Scholar by birth, its only hard work and strong
>>> determination that makes him master.
>>>
>>>
>>>
>>
>> --
>>
>> Garvit Sharma
>> github.com/garvitlnmiit/
>>
>> No Body is a Scholar by birth, its only hard work and strong
>> determination that makes him master.
>>
>
>
> --
>
> Garvit Sharma
> github.com/garvitlnmiit/
>
> No Body is a Scholar by birth, its only hard work and strong determination
> that makes him master.
>
Re: Exception while submitting jobs through Yarn
Posted by Garvit Sharma <ga...@gmail.com>.
Does someone has any idea how to get rid if the above parse exception while
submitting flink job to Yarn.
Already searched on the internet, could not find any solution to it.
Please help.
On Fri, Jun 15, 2018 at 9:15 AM Garvit Sharma <ga...@gmail.com> wrote:
> Thanks Chesnay, Now it is connecting to the Resource Manager but I am
> getting the below exception :
>
> 2018-06-15 09:12:44,812 FATAL org.apache.hadoop.conf.Configuration
> - error parsing conf core-default.xml
>
> javax.xml.parsers.ParserConfigurationException: Feature '
> http://apache.org/xml/features/xinclude' is not recognized.
>
> at
> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
> Source)
>
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737)
>
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696)
>
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
>
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1350)
>
> at
> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>
> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:584)
>
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:614)
>
> at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>
> at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>
> at
> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>
> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>
> 2018-06-15 09:12:44,825 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor
> - Error while getting queue information from YARN: null
>
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.yarn.util.Records
>
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:230)
>
> at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:497)
>
> at
> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>
> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>
> Please help.
>
> Thanks,
>
>
> On Thu, Jun 14, 2018 at 1:28 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> My gut feeling is that these classes must be present in jars in the /lib
>> directory. I don't think you can supply these with the submitted jar.
>> For a simple test, put your jar into the /lib folder before submitting it.
>>
>> On 14.06.2018 06:56, Garvit Sharma wrote:
>>
>> Can someone please tell why am I facing this?
>>
>> On Wed, Jun 13, 2018 at 10:33 PM Garvit Sharma <ga...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am using *flink-1.5.0-bin-hadoop27-scala_2.11 *to submit jobs through
>>> Yarn, but I am getting the below exception :
>>>
>>> java.lang.NoClassDefFoundError:
>>> com/sun/jersey/core/util/FeaturesAndProperties
>>>
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>>
>>> at
>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>>
>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>>
>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>>
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>>
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>>
>>> at java.security.AccessController.doPrivileged(Native Method)
>>>
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>>
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>>
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>
>>> at
>>> org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
>>>
>>> at
>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
>>>
>>> at
>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
>>>
>>> at
>>> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>>>
>>> at
>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.getClusterDescriptor(FlinkYarnSessionCli.java:966)
>>>
>>> at
>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createDescriptor(FlinkYarnSessionCli.java:269)
>>>
>>> at
>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:444)
>>>
>>> at
>>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:92)
>>>
>>> at
>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:221)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>
>>> at
>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>
>>> at
>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>
>>> at java.security.AccessController.doPrivileged(Native Method)
>>>
>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
>>>
>>> at
>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>
>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>
>>> Caused by: java.lang.ClassNotFoundException:
>>> com.sun.jersey.core.util.FeaturesAndProperties
>>>
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>>
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>>
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>
>>>
>>> Command : HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -m yarn-cluster
>>> -yd -yn 2 -ys 20 -yjm 10240 -ytm 10240 -yst -ynm test -yqu default -p 20
>>> test.jar
>>>
>>> The class *com/sun/jersey/core/util/FeaturesAndProperties* is already
>>> present in the test.jar so not sure why am I getting this exception.
>>>
>>> Please check and let me know.
>>>
>>> Thanks,
>>> --
>>>
>>> Garvit Sharma
>>> github.com/garvitlnmiit/
>>>
>>> No Body is a Scholar by birth, its only hard work and strong
>>> determination that makes him master.
>>>
>>
>>
>> --
>>
>> Garvit Sharma
>> github.com/garvitlnmiit/
>>
>> No Body is a Scholar by birth, its only hard work and strong
>> determination that makes him master.
>>
>>
>>
>
> --
>
> Garvit Sharma
> github.com/garvitlnmiit/
>
> No Body is a Scholar by birth, its only hard work and strong determination
> that makes him master.
>
--
Garvit Sharma
github.com/garvitlnmiit/
No Body is a Scholar by birth, its only hard work and strong determination
that makes him master.
Re: Exception while submitting jobs through Yarn
Posted by Garvit Sharma <ga...@gmail.com>.
Thanks Chesnay, Now it is connecting to the Resource Manager but I am
getting the below exception :
2018-06-15 09:12:44,812 FATAL org.apache.hadoop.conf.Configuration
- error parsing conf core-default.xml
javax.xml.parsers.ParserConfigurationException: Feature '
http://apache.org/xml/features/xinclude' is not recognized.
at
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
Source)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1350)
at
org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:584)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:614)
at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
at
org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
2018-06-15 09:12:44,825 WARN
org.apache.flink.yarn.AbstractYarnClusterDescriptor
- Error while getting queue information from YARN: null
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.hadoop.yarn.util.Records
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:230)
at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:497)
at
org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
Please help.
Thanks,
On Thu, Jun 14, 2018 at 1:28 PM Chesnay Schepler <ch...@apache.org> wrote:
> My gut feeling is that these classes must be present in jars in the /lib
> directory. I don't think you can supply these with the submitted jar.
> For a simple test, put your jar into the /lib folder before submitting it.
>
> On 14.06.2018 06:56, Garvit Sharma wrote:
>
> Can someone please tell why am I facing this?
>
> On Wed, Jun 13, 2018 at 10:33 PM Garvit Sharma <ga...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am using *flink-1.5.0-bin-hadoop27-scala_2.11 *to submit jobs through
>> Yarn, but I am getting the below exception :
>>
>> java.lang.NoClassDefFoundError:
>> com/sun/jersey/core/util/FeaturesAndProperties
>>
>> at java.lang.ClassLoader.defineClass1(Native Method)
>>
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>>
>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>>
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>>
>> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>>
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>>
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>
>> at
>> org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
>>
>> at
>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
>>
>> at
>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
>>
>> at
>> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>>
>> at
>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.getClusterDescriptor(FlinkYarnSessionCli.java:966)
>>
>> at
>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createDescriptor(FlinkYarnSessionCli.java:269)
>>
>> at
>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:444)
>>
>> at
>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:92)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:221)
>>
>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>
>> at
>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
>>
>> at
>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>
>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>
>> Caused by: java.lang.ClassNotFoundException:
>> com.sun.jersey.core.util.FeaturesAndProperties
>>
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>
>>
>> Command : HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -m yarn-cluster
>> -yd -yn 2 -ys 20 -yjm 10240 -ytm 10240 -yst -ynm test -yqu default -p 20
>> test.jar
>>
>> The class *com/sun/jersey/core/util/FeaturesAndProperties* is already
>> present in the test.jar so not sure why am I getting this exception.
>>
>> Please check and let me know.
>>
>> Thanks,
>> --
>>
>> Garvit Sharma
>> github.com/garvitlnmiit/
>>
>> No Body is a Scholar by birth, its only hard work and strong
>> determination that makes him master.
>>
>
>
> --
>
> Garvit Sharma
> github.com/garvitlnmiit/
>
> No Body is a Scholar by birth, its only hard work and strong determination
> that makes him master.
>
>
>
--
Garvit Sharma
github.com/garvitlnmiit/
No Body is a Scholar by birth, its only hard work and strong determination
that makes him master.
Re: Exception while submitting jobs through Yarn
Posted by Chesnay Schepler <ch...@apache.org>.
My gut feeling is that these classes must be present in jars in the /lib
directory. I don't think you can supply these with the submitted jar.
For a simple test, put your jar into the /lib folder before submitting it.
On 14.06.2018 06:56, Garvit Sharma wrote:
> Can someone please tell why am I facing this?
>
> On Wed, Jun 13, 2018 at 10:33 PM Garvit Sharma <garvits45@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi,
>
> I am using *flink-1.5.0-bin-hadoop27-scala_2.11 *to submit jobs
> through Yarn, but I am getting the below exception :
>
> java.lang.NoClassDefFoundError:
> com/sun/jersey/core/util/FeaturesAndProperties
>
> at java.lang.ClassLoader.defineClass1(Native Method)
>
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
> at
> org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
>
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
>
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
>
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.getClusterDescriptor(FlinkYarnSessionCli.java:966)
>
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createDescriptor(FlinkYarnSessionCli.java:269)
>
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:444)
>
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:92)
>
> at
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:221)
>
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
>
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>
> Caused by: java.lang.ClassNotFoundException:
> com.sun.jersey.core.util.FeaturesAndProperties
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
>
>
> Command : HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -m
> yarn-cluster -yd -yn 2 -ys 20 -yjm 10240 -ytm 10240 -yst -ynm test
> -yqu default -p 20 test.jar
>
> The class *com/sun/jersey/core/util/FeaturesAndProperties* is
> already present in the test.jar so not sure why am I getting this
> exception.
>
> Please check and let me know.
>
> Thanks,
> --
>
> Garvit Sharma
> github.com/garvitlnmiit/ <http://github.com/garvitlnmiit/>
>
> No Body is a Scholar by birth, its only hard work and strong
> determination that makes him master.
>
>
>
> --
>
> Garvit Sharma
> github.com/garvitlnmiit/ <http://github.com/garvitlnmiit/>
>
> No Body is a Scholar by birth, its only hard work and strong
> determination that makes him master.
Re: Exception while submitting jobs through Yarn
Posted by Garvit Sharma <ga...@gmail.com>.
Can someone please tell why am I facing this?
On Wed, Jun 13, 2018 at 10:33 PM Garvit Sharma <ga...@gmail.com> wrote:
> Hi,
>
> I am using *flink-1.5.0-bin-hadoop27-scala_2.11 *to submit jobs through
> Yarn, but I am getting the below exception :
>
> java.lang.NoClassDefFoundError:
> com/sun/jersey/core/util/FeaturesAndProperties
>
> at java.lang.ClassLoader.defineClass1(Native Method)
>
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
>
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
> at
> org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)
>
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)
>
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)
>
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.getClusterDescriptor(FlinkYarnSessionCli.java:966)
>
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createDescriptor(FlinkYarnSessionCli.java:269)
>
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:444)
>
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createClusterDescriptor(FlinkYarnSessionCli.java:92)
>
> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:221)
>
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
>
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>
> Caused by: java.lang.ClassNotFoundException:
> com.sun.jersey.core.util.FeaturesAndProperties
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
>
> Command : HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -m yarn-cluster
> -yd -yn 2 -ys 20 -yjm 10240 -ytm 10240 -yst -ynm test -yqu default -p 20
> test.jar
>
> The class *com/sun/jersey/core/util/FeaturesAndProperties* is already
> present in the test.jar so not sure why am I getting this exception.
>
> Please check and let me know.
>
> Thanks,
> --
>
> Garvit Sharma
> github.com/garvitlnmiit/
>
> No Body is a Scholar by birth, its only hard work and strong determination
> that makes him master.
>
--
Garvit Sharma
github.com/garvitlnmiit/
No Body is a Scholar by birth, its only hard work and strong determination
that makes him master.