You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Mikhail Pryakhin <m....@cleverdata.ru> on 2017/05/22 15:12:11 UTC
yarnship option
Hi all!
I'm playing with flink streaming job on yarn cluster. The job consumes events from kafka and prints them to the standard out.
The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is passed via the --yarnship option.
Here is the way I run the job:
export HADOOP_USER_NAME=hdfs; \
export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
/opt/flink-1.2.1/bin/flink run \
-yst \
-yt /home/user/job-libs \
-m yarn-cluster \
-yn 3 \
-c com.flink.Test \
flink-test_2.11-1.0.0-SNAPSHOT.jar
Finally the job fails complaing that it can't find the class:
java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09
I looked through the jobmanager.log and found that the flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the classpath:
2017-05-22 17:41:59,637 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Classpath: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another jars...>
Could please help to figure out why the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't be loaded inspite of the jar containing this class was added to the classpath
----
Mike Pryakhin
Re: yarnship option
Posted by Mikhail Pryakhin <m....@cleverdata.ru>.
Hi Robert,
Thanks for your advice but this option doesn’t work for us because our job depends on third party jars with transitive dependencies to the same jar but of different versions and it hardly possible to say which version will participate in the fat jar..
Anyway thanks a lot for your help.
——
Mike Pryakhin
> On 23 May 2017, at 16:04, Robert Metzger <rm...@apache.org> wrote:
>
> Hi Mike,
>
> I would recommend you to build a "fat jar" containing your application code and all required dependencies.
>
> On Tue, May 23, 2017 at 10:33 AM, Mikhail Pryakhin <m.pryahin@cleverdata.ru <ma...@cleverdata.ru>> wrote:
> Hi Robert!
>
> You were right.. it was my fault, I didn’t copy all required dependencies for the flink-connector-kafka. That made impossible to load the class.
>
> Everything works as expected but only if I put required dependencies inside the flink lib folder, but when dependencies are located in a different folder which is passed in the yarnship command line option then they seem not to be put on the TM's classpath, and the job fails with ClassNotFoundException.
> Is there anything else I need to provide to make libraries in that folder to be on the TM's classpath?
>
> Many thanks,
>
> ——
>
> Mike Pryakhin
>
>
>> On 22 May 2017, at 23:06, Mikhail Pryakhin <m.pryahin@cleverdata.ru <ma...@cleverdata.ru>> wrote:
>>
>> Hi Robert!
>> Thanks a lot for your reply!
>>
>> >Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
>>
>> The jar does contain the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class (The contents of the jar file is listed below)
>>
>> > Also, were there any previous Kafka09 related exceptions in the log??
>> No, it was the very first exception… You can find log file attached.
>>
>>
>> the flink-connector-kafka-0.9_2.11-1.2.1.jar contains the following:
>>
>> root@host:~# unzip -l job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
>> Archive: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
>> Length Date Time Name
>> --------- ---------- ----- ----
>> 0 2017-04-11 01:59 META-INF/
>> 443 2017-04-11 01:59 META-INF/MANIFEST.MF
>> 1451 2017-04-11 01:59 META-INF/DEPENDENCIES
>> 11358 2017-04-11 01:59 META-INF/LICENSE
>> 182 2017-04-11 01:59 META-INF/NOTICE
>> 0 2017-04-11 01:59 org/
>> 0 2017-04-11 01:59 org/apache/
>> 0 2017-04-11 01:59 org/apache/flink/
>> 0 2017-04-11 01:59 org/apache/flink/streaming/
>> 0 2017-04-11 01:59 org/apache/flink/streaming/connectors/
>> 0 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/
>> 11554 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class
>> 2603 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSink.class
>> 4463 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer09.class
>> 2693 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/Kafka09TableSource.class
>> 0 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/
>> 1125 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerCallBridge.class
>> 570 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/Handover$WakeupException.class
>> 10268 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.class
>> 570 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/Handover$ClosedException.class
>> 3018 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/Handover.class
>> 11854 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.class
>> 2182 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$CommitCallback.class
>> 311 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$1.class
>> 2329 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSource.class
>> 0 2017-04-11 01:59 META-INF/maven/
>> 0 2017-04-11 01:59 META-INF/maven/org.apache.flink/
>> 0 2017-04-11 01:59 META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/
>> 6039 2017-04-11 01:59 META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.xml
>> 131 2017-04-11 01:59 META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.properties
>> 0 2017-04-11 01:59 META-INF/maven/org.apache.flink/force-shading/
>> 3285 2017-04-11 01:59 META-INF/maven/org.apache.flink/force-shading/pom.xml
>> 114 2017-04-11 01:59 META-INF/maven/org.apache.flink/force-shading/pom.properties
>> --------- -------
>> 76543 33 files
>>
>>
>>
>>
>> <flink-root-client-dmpkit-dev-dn1.log>
>>
>>
>> ------
>> Mike Pryakhin
>>
>>
>>
>>> On 22 May 2017, at 22:14, Robert Metzger <rmetzger@apache.org <ma...@apache.org>> wrote:
>>>
>>> Hi,
>>>
>>> this issue is unexpected :) Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
>>> Also, were there any previous Kafka09 related exceptions in the log??
>>>
>>> From this SO answer, it seems that this is not really the classical classNotFoundException, but a bit differenT: https://stackoverflow.com/a/5756989/568695 <https://stackoverflow.com/a/5756989/568695>
>>>
>>> On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <m.pryahin@cleverdata.ru <ma...@cleverdata.ru>> wrote:
>>> Hi all!
>>>
>>> I'm playing with flink streaming job on yarn cluster. The job consumes events from kafka and prints them to the standard out.
>>> The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is passed via the --yarnship option.
>>> Here is the way I run the job:
>>>
>>> export HADOOP_USER_NAME=hdfs; \
>>> export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
>>> /opt/flink-1.2.1/bin/flink run \
>>> -yst \
>>> -yt /home/user/job-libs \
>>> -m yarn-cluster \
>>> -yn 3 \
>>> -c com.flink.Test \
>>> flink-test_2.11-1.0.0-SNAPSHOT.jar
>>>
>>> Finally the job fails complaing that it can't find the class:
>>> java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09
>>>
>>> I looked through the jobmanager.log and found that the flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the classpath:
>>>
>>> 2017-05-22 17:41:59,637 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Classpath: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another jars...>
>>>
>>> Could please help to figure out why the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't be loaded inspite of the jar containing this class was added to the classpath
>>>
>>> ----
>>> Mike Pryakhin
>>>
>>
>
>
Re: yarnship option
Posted by Robert Metzger <rm...@apache.org>.
Hi Mike,
I would recommend you to build a "fat jar" containing your application code
and all required dependencies.
On Tue, May 23, 2017 at 10:33 AM, Mikhail Pryakhin <m....@cleverdata.ru>
wrote:
> Hi Robert!
>
> You were right.. it was my fault, I didn’t copy all required dependencies
> for the flink-connector-kafka. That made impossible to load the class.
>
> Everything works as expected but only if I put required dependencies
> inside the flink lib folder, but when dependencies are located in a
> different folder which is passed in the yarnship command line option then
> they seem not to be put on the TM's classpath, and the job fails with
> ClassNotFoundException.
> Is there anything else I need to provide to make libraries in that folder
> to be on the TM's classpath?
>
> Many thanks,
>
> ——
>
> Mike Pryakhin
>
>
> On 22 May 2017, at 23:06, Mikhail Pryakhin <m....@cleverdata.ru>
> wrote:
>
> Hi Robert!
> Thanks a lot for your reply!
>
> >Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
> contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09
> ?
>
> The jar does contain the class org/apache/flink/streaming/
> connectors/kafka/FlinkKafkaConsumer09.class (The contents of the jar file
> is listed below)
>
> > Also, were there any previous Kafka09 related exceptions in the log??
> No, it was the very first exception… You can find log file attached.
>
>
> the flink-connector-kafka-0.9_2.11-1.2.1.jar contains the following:
>
> root@host:~# unzip -l job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
> Archive: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
> Length Date Time Name
> --------- ---------- ----- ----
> 0 2017-04-11 01:59 META-INF/
> 443 2017-04-11 01:59 META-INF/MANIFEST.MF
> 1451 2017-04-11 01:59 META-INF/DEPENDENCIES
> 11358 2017-04-11 01:59 META-INF/LICENSE
> 182 2017-04-11 01:59 META-INF/NOTICE
> 0 2017-04-11 01:59 org/
> 0 2017-04-11 01:59 org/apache/
> 0 2017-04-11 01:59 org/apache/flink/
> 0 2017-04-11 01:59 org/apache/flink/streaming/
> 0 2017-04-11 01:59 org/apache/flink/streaming/connectors/
> 0 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/
> 11554 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/
> FlinkKafkaConsumer09.class
> 2603 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/
> Kafka09JsonTableSink.class
> 4463 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/
> FlinkKafkaProducer09.class
> 2693 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/
> Kafka09TableSource.class
> 0 2017-04-11 01:59 org/apache/flink/streaming/
> connectors/kafka/internal/
> 1125 2017-04-11 01:59 org/apache/flink/streaming/
> connectors/kafka/internal/KafkaConsumerCallBridge.class
> 570 2017-04-11 01:59 org/apache/flink/streaming/
> connectors/kafka/internal/Handover$WakeupException.class
> 10268 2017-04-11 01:59 org/apache/flink/streaming/
> connectors/kafka/internal/KafkaConsumerThread.class
> 570 2017-04-11 01:59 org/apache/flink/streaming/
> connectors/kafka/internal/Handover$ClosedException.class
> 3018 2017-04-11 01:59 org/apache/flink/streaming/
> connectors/kafka/internal/Handover.class
> 11854 2017-04-11 01:59 org/apache/flink/streaming/
> connectors/kafka/internal/Kafka09Fetcher.class
> 2182 2017-04-11 01:59 org/apache/flink/streaming/
> connectors/kafka/internal/KafkaConsumerThread$CommitCallback.class
> 311 2017-04-11 01:59 org/apache/flink/streaming/
> connectors/kafka/internal/KafkaConsumerThread$1.class
> 2329 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/
> Kafka09JsonTableSource.class
> 0 2017-04-11 01:59 META-INF/maven/
> 0 2017-04-11 01:59 META-INF/maven/org.apache.flink/
> 0 2017-04-11 01:59 META-INF/maven/org.apache.
> flink/flink-connector-kafka-0.9_2.11/
> 6039 2017-04-11 01:59 META-INF/maven/org.apache.
> flink/flink-connector-kafka-0.9_2.11/pom.xml
> 131 2017-04-11 01:59 META-INF/maven/org.apache.
> flink/flink-connector-kafka-0.9_2.11/pom.properties
> 0 2017-04-11 01:59 META-INF/maven/org.apache.
> flink/force-shading/
> 3285 2017-04-11 01:59 META-INF/maven/org.apache.
> flink/force-shading/pom.xml
> 114 2017-04-11 01:59 META-INF/maven/org.apache.
> flink/force-shading/pom.properties
> --------- -------
> 76543 33 files
>
>
>
>
> <flink-root-client-dmpkit-dev-dn1.log>
>
>
> ------
> Mike Pryakhin
>
>
>
> On 22 May 2017, at 22:14, Robert Metzger <rm...@apache.org> wrote:
>
> Hi,
>
> this issue is unexpected :) Can you double check if the
> job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains
> org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
> Also, were there any previous Kafka09 related exceptions in the log??
>
> From this SO answer, it seems that this is not really the classical
> classNotFoundException, but a bit differenT: https://
> stackoverflow.com/a/5756989/568695
>
> On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <m.pryahin@cleverdata.ru
> > wrote:
>
>> Hi all!
>>
>> I'm playing with flink streaming job on yarn cluster. The job consumes
>> events from kafka and prints them to the standard out.
>> The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is
>> passed via the --yarnship option.
>> Here is the way I run the job:
>>
>> export HADOOP_USER_NAME=hdfs; \
>> export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
>> /opt/flink-1.2.1/bin/flink run \
>> -yst \
>> -yt /home/user/job-libs \
>> -m yarn-cluster \
>> -yn 3 \
>> -c com.flink.Test \
>> flink-test_2.11-1.0.0-SNAPSHOT.jar
>>
>> Finally the job fails complaing that it can't find the class:
>> java.lang.NoClassDefFoundError: org/apache/flink/streaming/con
>> nectors/kafka/FlinkKafkaConsumer09
>>
>> I looked through the jobmanager.log and found that the
>> flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the
>> classpath:
>>
>> 2017-05-22 17:41:59,637 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
>> - Classpath: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another
>> jars...>
>>
>> Could please help to figure out why the class
>> org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't
>> be loaded inspite of the jar containing this class was added to the
>> classpath
>>
>> ----
>> Mike Pryakhin
>
>
>
>
>
Re: yarnship option
Posted by Mikhail Pryakhin <m....@cleverdata.ru>.
Hi Robert!
You were right.. it was my fault, I didn’t copy all required dependencies for the flink-connector-kafka. That made impossible to load the class.
Everything works as expected but only if I put required dependencies inside the flink lib folder, but when dependencies are located in a different folder which is passed in the yarnship command line option then they seem not to be put on the TM's classpath, and the job fails with ClassNotFoundException.
Is there anything else I need to provide to make libraries in that folder to be on the TM's classpath?
Many thanks,
——
Mike Pryakhin
> On 22 May 2017, at 23:06, Mikhail Pryakhin <m....@cleverdata.ru> wrote:
>
> Hi Robert!
> Thanks a lot for your reply!
>
> >Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
>
> The jar does contain the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class (The contents of the jar file is listed below)
>
> > Also, were there any previous Kafka09 related exceptions in the log??
> No, it was the very first exception… You can find log file attached.
>
>
> the flink-connector-kafka-0.9_2.11-1.2.1.jar contains the following:
>
> root@host:~# unzip -l job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
> Archive: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
> Length Date Time Name
> --------- ---------- ----- ----
> 0 2017-04-11 01:59 META-INF/
> 443 2017-04-11 01:59 META-INF/MANIFEST.MF
> 1451 2017-04-11 01:59 META-INF/DEPENDENCIES
> 11358 2017-04-11 01:59 META-INF/LICENSE
> 182 2017-04-11 01:59 META-INF/NOTICE
> 0 2017-04-11 01:59 org/
> 0 2017-04-11 01:59 org/apache/
> 0 2017-04-11 01:59 org/apache/flink/
> 0 2017-04-11 01:59 org/apache/flink/streaming/
> 0 2017-04-11 01:59 org/apache/flink/streaming/connectors/
> 0 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/
> 11554 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class
> 2603 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSink.class
> 4463 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer09.class
> 2693 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/Kafka09TableSource.class
> 0 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/
> 1125 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerCallBridge.class
> 570 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/Handover$WakeupException.class
> 10268 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.class
> 570 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/Handover$ClosedException.class
> 3018 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/Handover.class
> 11854 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.class
> 2182 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$CommitCallback.class
> 311 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$1.class
> 2329 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSource.class
> 0 2017-04-11 01:59 META-INF/maven/
> 0 2017-04-11 01:59 META-INF/maven/org.apache.flink/
> 0 2017-04-11 01:59 META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/
> 6039 2017-04-11 01:59 META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.xml
> 131 2017-04-11 01:59 META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.properties
> 0 2017-04-11 01:59 META-INF/maven/org.apache.flink/force-shading/
> 3285 2017-04-11 01:59 META-INF/maven/org.apache.flink/force-shading/pom.xml
> 114 2017-04-11 01:59 META-INF/maven/org.apache.flink/force-shading/pom.properties
> --------- -------
> 76543 33 files
>
>
>
>
> <flink-root-client-dmpkit-dev-dn1.log>
>
>
> ------
> Mike Pryakhin
>
>
>
>> On 22 May 2017, at 22:14, Robert Metzger <rmetzger@apache.org <ma...@apache.org>> wrote:
>>
>> Hi,
>>
>> this issue is unexpected :) Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
>> Also, were there any previous Kafka09 related exceptions in the log??
>>
>> From this SO answer, it seems that this is not really the classical classNotFoundException, but a bit differenT: https://stackoverflow.com/a/5756989/568695 <https://stackoverflow.com/a/5756989/568695>
>>
>> On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <m.pryahin@cleverdata.ru <ma...@cleverdata.ru>> wrote:
>> Hi all!
>>
>> I'm playing with flink streaming job on yarn cluster. The job consumes events from kafka and prints them to the standard out.
>> The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is passed via the --yarnship option.
>> Here is the way I run the job:
>>
>> export HADOOP_USER_NAME=hdfs; \
>> export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
>> /opt/flink-1.2.1/bin/flink run \
>> -yst \
>> -yt /home/user/job-libs \
>> -m yarn-cluster \
>> -yn 3 \
>> -c com.flink.Test \
>> flink-test_2.11-1.0.0-SNAPSHOT.jar
>>
>> Finally the job fails complaing that it can't find the class:
>> java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09
>>
>> I looked through the jobmanager.log and found that the flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the classpath:
>>
>> 2017-05-22 17:41:59,637 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Classpath: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another jars...>
>>
>> Could please help to figure out why the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't be loaded inspite of the jar containing this class was added to the classpath
>>
>> ----
>> Mike Pryakhin
>>
>
Re: yarnship option
Posted by Mikhail Pryakhin <m....@cleverdata.ru>.
Hi Robert!
Thanks a lot for your reply!
>Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
The jar does contain the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class (The contents of the jar file is listed below)
> Also, were there any previous Kafka09 related exceptions in the log??
No, it was the very first exception… You can find log file attached.
the flink-connector-kafka-0.9_2.11-1.2.1.jar contains the following:
root@host:~# unzip -l job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
Archive: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar
Length Date Time Name
--------- ---------- ----- ----
0 2017-04-11 01:59 META-INF/
443 2017-04-11 01:59 META-INF/MANIFEST.MF
1451 2017-04-11 01:59 META-INF/DEPENDENCIES
11358 2017-04-11 01:59 META-INF/LICENSE
182 2017-04-11 01:59 META-INF/NOTICE
0 2017-04-11 01:59 org/
0 2017-04-11 01:59 org/apache/
0 2017-04-11 01:59 org/apache/flink/
0 2017-04-11 01:59 org/apache/flink/streaming/
0 2017-04-11 01:59 org/apache/flink/streaming/connectors/
0 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/
11554 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09.class
2603 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSink.class
4463 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer09.class
2693 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/Kafka09TableSource.class
0 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/
1125 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerCallBridge.class
570 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/Handover$WakeupException.class
10268 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.class
570 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/Handover$ClosedException.class
3018 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/Handover.class
11854 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.class
2182 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$CommitCallback.class
311 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread$1.class
2329 2017-04-11 01:59 org/apache/flink/streaming/connectors/kafka/Kafka09JsonTableSource.class
0 2017-04-11 01:59 META-INF/maven/
0 2017-04-11 01:59 META-INF/maven/org.apache.flink/
0 2017-04-11 01:59 META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/
6039 2017-04-11 01:59 META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.xml
131 2017-04-11 01:59 META-INF/maven/org.apache.flink/flink-connector-kafka-0.9_2.11/pom.properties
0 2017-04-11 01:59 META-INF/maven/org.apache.flink/force-shading/
3285 2017-04-11 01:59 META-INF/maven/org.apache.flink/force-shading/pom.xml
114 2017-04-11 01:59 META-INF/maven/org.apache.flink/force-shading/pom.properties
--------- -------
76543 33 files
------
Mike Pryakhin
> On 22 May 2017, at 22:14, Robert Metzger <rm...@apache.org> wrote:
>
> Hi,
>
> this issue is unexpected :) Can you double check if the job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
> Also, were there any previous Kafka09 related exceptions in the log??
>
> From this SO answer, it seems that this is not really the classical classNotFoundException, but a bit differenT: https://stackoverflow.com/a/5756989/568695 <https://stackoverflow.com/a/5756989/568695>
>
> On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <m.pryahin@cleverdata.ru <ma...@cleverdata.ru>> wrote:
> Hi all!
>
> I'm playing with flink streaming job on yarn cluster. The job consumes events from kafka and prints them to the standard out.
> The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is passed via the --yarnship option.
> Here is the way I run the job:
>
> export HADOOP_USER_NAME=hdfs; \
> export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
> /opt/flink-1.2.1/bin/flink run \
> -yst \
> -yt /home/user/job-libs \
> -m yarn-cluster \
> -yn 3 \
> -c com.flink.Test \
> flink-test_2.11-1.0.0-SNAPSHOT.jar
>
> Finally the job fails complaing that it can't find the class:
> java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09
>
> I looked through the jobmanager.log and found that the flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the classpath:
>
> 2017-05-22 17:41:59,637 INFO org.apache.flink.yarn.YarnApplicationMasterRunner - Classpath: job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar:<another jars...>
>
> Could please help to figure out why the class org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 can't be loaded inspite of the jar containing this class was added to the classpath
>
> ----
> Mike Pryakhin
>
Re: yarnship option
Posted by Robert Metzger <rm...@apache.org>.
Hi,
this issue is unexpected :) Can you double check if the
job-libs/flink-connector-kafka-0.9_2.11-1.2.1.jar contains
org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer09 ?
Also, were there any previous Kafka09 related exceptions in the log??
From this SO answer, it seems that this is not really the classical
classNotFoundException, but a bit differenT:
https://stackoverflow.com/a/5756989/568695
On Mon, May 22, 2017 at 5:12 PM, Mikhail Pryakhin <m....@cleverdata.ru>
wrote:
> Hi all!
>
> I'm playing with flink streaming job on yarn cluster. The job consumes
> events from kafka and prints them to the standard out.
> The job uses flink-connector-kafka-0.9_2.11-1.2.1.jar library that is
> passed via the --yarnship option.
> Here is the way I run the job:
>
> export HADOOP_USER_NAME=hdfs; \
> export HADOOP_CONF_DIR=/etc/hadoop/conf/; \
> /opt/flink-1.2.1/bin/flink run \
> -yst \
> -yt /home/user/job-libs \
> -m yarn-cluster \
> -yn 3 \
> -c com.flink.Test \
> flink-test_2.11-1.0.0-SNAPSHOT.jar
>
> Finally the job fails complaing that it can't find the class:
> java.lang.NoClassDefFoundError: org/apache/flink/streaming/
> connectors/kafka/FlinkKafkaConsumer09
>
> I looked through the jobmanager.log and found that the
> flink-connector-kafka-0.9_2.11-1.2.1.jar library is added to the
> classpath:
>
> 2017-05-22 17:41:59,637 INFO org.apache.flink.yarn.YarnApplicationMasterRunner
> - Classpath: job-libs/flink-connector-
> kafka-0.9_2.11-1.2.1.jar:<another jars...>
>
> Could please help to figure out why the class org/apache/flink/streaming/
> connectors/kafka/FlinkKafkaConsumer09 can't be loaded inspite of the jar
> containing this class was added to the classpath
>
> ----
> Mike Pryakhin