You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by hjw <10...@qq.com> on 2022/09/12 17:58:20 UTC
A classloading question about submitting Flink jobs on Yarn-Perjob Mode
When I submit a job to yarn on Yarn-perjob Mode.An error occurred during the client-side generation of the Jobgraph submitd to yarn.
Error:
java.lang.NoClassDefFoundError：org/apache/orc/PhysicalWriter
Because the cluster is public, there is already related flink-orc_2.12-1.14.0-csa1.6.1.0.jar package in the /opt/flink/lib directory.
However ,this class is included by my jar. This class is provided by orc-core package. I have packaged it in my jar.
After my attempts, the following measures can solve my problem.
1.remove the related flink-orc_2.12-1.14.0-csa1.6.1.0.jar from /opt/flink/lib directory and packaged it to my jar.
2.put the orc-core to /opt/flink/lib directory.
However, I would like to know why an error occurs when placing the flink-orc_2.12-1.14.0-csa1.6.1.0.jar package in the /opt/flink/lib directory and packaging orc-core into the my jar.
Env：
Flink version: flink 1.14.0
Best,
Hjw
Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode
Posted by yu'an huang <h....@gmail.com>.
I guess there may be a class conflict between your user jar and Flink lib. For the error: java.lang.NoClassDefFoundError, it may caused Exception happening when Initializing a class. I suggest you set the log level to DEBUG and send the client log here. Let’s look whether there are any new finding in the client log.
Best,
Yuan
> On 13 Sep 2022, at 1:55 PM, zhanghao.chen@outlook.com wrote:
>
> @h.yuan667@gmail.com <ma...@gmail.com> Any thoughts on this?
>
> Best,
> Zhanghao Chen
> From: hjw <10...@qq.com>
> Sent: Tuesday, September 13, 2022 11:24
> To: zhanghao.chen <zh...@outlook.com>; user <us...@flink.apache.org>
> Subject: Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode
>
> Hi,
>
> The yarn.classpath.include-user-jar parameter is shown as yarn.per-job-cluster.include-user-jar parameter in Flink 1.14.
> I have tried DISABLED、FIRST、LAST、ORDER .But the error still happened.
> Best,
> Hjw
>
>
>
> ------------------ Original ------------------
> From: "zhanghao.chen" <zh...@outlook.com>;
> Date: Tue, Sep 13, 2022 09:42 AM
> To: "hjw"<10...@flink.apache.org>;
> Subject: Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode
>
> Hi,
>
> Did you set any additional classloading-related configs (esp. the yarn.classpath.include-user-jar parameter)?
>
> Best,
> Zhanghao Chen
> From: hjw <10...@qq.com>
> Sent: Tuesday, September 13, 2022 1:58
> To: user <us...@flink.apache.org>
> Subject: A classloading question about submitting Flink jobs on Yarn-Perjob Mode
>
> When I submit a job to yarn on Yarn-perjob Mode.An error occurred during the client-side generation of the Jobgraph submitd to yarn.
>
> Error:
> java.lang.NoClassDefFoundError：org/apache/orc/PhysicalWriter
>
> Because the cluster is public, there is already related flink-orc_2.12-1.14.0-csa1.6.1.0.jar package in the /opt/flink/lib directory.
>
> However ,this class is included by my jar. This class is provided by orc-core package. I have packaged it in my jar.
>
> After my attempts, the following measures can solve my problem.
> 1.remove the related flink-orc_2.12-1.14.0-csa1.6.1.0.jar from /opt/flink/lib directory and packaged it to my jar.
> 2.put the orc-core to /opt/flink/lib directory.
>
> However, I would like to know why an error occurs when placing the flink-orc_2.12-1.14.0-csa1.6.1.0.jar package in the /opt/flink/lib directory and packaging orc-core into the my jar.
>
>
> Env：
> Flink version: flink 1.14.0
>
> Best,
> Hjw
Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode
Posted by zh...@outlook.com.
@h.yuan667@gmail.com<ma...@gmail.com> Any thoughts on this?
Best,
Zhanghao Chen
________________________________
From: hjw <10...@qq.com>
Sent: Tuesday, September 13, 2022 11:24
To: zhanghao.chen <zh...@outlook.com>; user <us...@flink.apache.org>
Subject: Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode
Hi,
The yarn.classpath.include-user-jar parameter is shown as yarn.per-job-cluster.include-user-jar parameter in Flink 1.14.
I have tried DISABLED、FIRST、LAST、ORDER .But the error still happened.
________________________________
Best,
Hjw
------------------ Original ------------------
From: "zhanghao.chen" <zh...@outlook.com>;
Date: Tue, Sep 13, 2022 09:42 AM
To: "hjw"<10...@flink.apache.org>;
Subject: Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode
Hi,
Did you set any additional classloading-related configs (esp. the yarn.classpath.include-user-jar parameter)?
Best,
Zhanghao Chen
________________________________
From: hjw <10...@qq.com>
Sent: Tuesday, September 13, 2022 1:58
To: user <us...@flink.apache.org>
Subject: A classloading question about submitting Flink jobs on Yarn-Perjob Mode
When I submit a job to yarn on Yarn-perjob Mode.An error occurred during the client-side generation of the Jobgraph submitd to yarn.
Error:
java.lang.NoClassDefFoundError：org/apache/orc/PhysicalWriter
Because the cluster is public, there is already related flink-orc_2.12-1.14.0-csa1.6.1.0.jar package in the /opt/flink/lib directory.
However ,this class is included by my jar. This class is provided by orc-core package. I have packaged it in my jar.
After my attempts, the following measures can solve my problem.
1.remove the related flink-orc_2.12-1.14.0-csa1.6.1.0.jar from /opt/flink/lib directory and packaged it to my jar.
2.put the orc-core to /opt/flink/lib directory.
However, I would like to know why an error occurs when placing the flink-orc_2.12-1.14.0-csa1.6.1.0.jar package in the /opt/flink/lib directory and packaging orc-core into the my jar.
Env：
Flink version: flink 1.14.0
________________________________
Best,
Hjw
Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode
Posted by hjw <10...@qq.com>.
Hi,
The yarn.classpath.include-user-jar parameter is shown as yarn.per-job-cluster.include-user-jar parameter in Flink 1.14.
I have tried DISABLED、FIRST、LAST、ORDER .But the error still happened.
Best,
Hjw
------------------ Original ------------------
From: "zhanghao.chen" <zhanghao.chen@outlook.com>;
Date: Tue, Sep 13, 2022 09:42 AM
To: "hjw"<1010445050@qq.com>;"user"<user@flink.apache.org>;
Subject: Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode
Hi,
Did you set any additional classloading-related configs (esp. the yarn.classpath.include-user-jar parameter)?
Best,
Zhanghao Chen
From: hjw <1010445050@qq.com>
Sent: Tuesday, September 13, 2022 1:58
To: user <user@flink.apache.org>
Subject: A classloading question about submitting Flink jobs on Yarn-Perjob Mode
When I submit a job to yarn on Yarn-perjob Mode.An error occurred during the client-side generation of the Jobgraph submitd to yarn.
Error:
java.lang.NoClassDefFoundError：org/apache/orc/PhysicalWriter
Because the cluster is public, there is already related flink-orc_2.12-1.14.0-csa1.6.1.0.jar package in the /opt/flink/lib directory.
However ,this class is included by my jar. This class is provided by orc-core package. I have packaged it in my jar.
After my attempts, the following measures can solve my problem.
1.remove the related flink-orc_2.12-1.14.0-csa1.6.1.0.jar from /opt/flink/lib directory and packaged it to my jar.
2.put the orc-core to /opt/flink/lib directory.
However, I would like to know why an error occurs when placing the flink-orc_2.12-1.14.0-csa1.6.1.0.jar package in the /opt/flink/lib directory and packaging orc-core into the my jar.
Env：
Flink version: flink 1.14.0
Best,
Hjw
Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode
Posted by zh...@outlook.com.
Hi,
Did you set any additional classloading-related configs (esp. the yarn.classpath.include-user-jar parameter)?
Best,
Zhanghao Chen
________________________________
From: hjw <10...@qq.com>
Sent: Tuesday, September 13, 2022 1:58
To: user <us...@flink.apache.org>
Subject: A classloading question about submitting Flink jobs on Yarn-Perjob Mode
When I submit a job to yarn on Yarn-perjob Mode.An error occurred during the client-side generation of the Jobgraph submitd to yarn.
Error:
java.lang.NoClassDefFoundError：org/apache/orc/PhysicalWriter
Because the cluster is public, there is already related flink-orc_2.12-1.14.0-csa1.6.1.0.jar package in the /opt/flink/lib directory.
However ,this class is included by my jar. This class is provided by orc-core package. I have packaged it in my jar.
After my attempts, the following measures can solve my problem.
1.remove the related flink-orc_2.12-1.14.0-csa1.6.1.0.jar from /opt/flink/lib directory and packaged it to my jar.
2.put the orc-core to /opt/flink/lib directory.
However, I would like to know why an error occurs when placing the flink-orc_2.12-1.14.0-csa1.6.1.0.jar package in the /opt/flink/lib directory and packaging orc-core into the my jar.
Env：
Flink version: flink 1.14.0
________________________________
Best,
Hjw