You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Kiran Kumar Kathe <ki...@gmail.com> on 2023/03/28 09:43:12 UTC
Jobs running in flink are not viable in YARN resource manager WEB UI
When I submit jobs to flink using version 1.9.3 those jobs are visible in
YARN resource manager Web UI , but when I use flink version 1.10.1 jobs
running in flink are not visible in the Web UI , So in order to check where
the issue is raising , I started replacing flink 1.10.1 folders with
flink 1.9.3 then , I get to know that lib folder can one jar file named
flink_dist which is responsible , to make the jobs running in the flink
visible in YARN resource manager. So my question here is Can I use the
flink_dist jar of version 1.9.3 in place of flink_dist 1.10.1 , if not can
anyone please give me clarity what configuration changes should i do in
order to make jobs running visible in YARN resource manager Web UI with
Flink version 1.10.1. [ I observed that while I am using flink 1.9.3 it is
starting a yarn-session and I can see logs related to that and when using
flink 1.10.1 there are no logs related to yarn-session are visible.]
Re: Jobs running in flink are not viable in YARN resource manager WEB UI
Posted by Kiran Kumar Kathe <ki...@gmail.com>.
This is the screenshot when I run flink job with flink 1.10.1 and the
terminal looks like
On Wed, 29 Mar 2023 at 11:19, Weihua Hu <hu...@gmail.com> wrote:
> I see. So the screenshots are using version 1.9.3. What does version
> 1.10.1 show?
>
> Best,
> Weihua
>
>
> On Wed, Mar 29, 2023 at 12:15 PM Kiran Kumar Kathe <
> kirankumarkathe123@gmail.com> wrote:
>
>> I am using yarn-cluster mode and command as ./flink run -m yarn-cluster
>> /home/kirankathe/s3checkup/original-demo-helloworld-0.0.1-SNAPSHOT.jar
>> and it is visible in the yarn UI like this as completed and the terminal
>> looks as in the screenshots attached
>>
>> On Wed, 29 Mar 2023 at 09:31, Weihua Hu <hu...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>>
>>>> jobs are visible in YARN resource manager Web UI
>>>
>>> Which mode are you using to submit a job to YARN? Session mode or Perjob
>>> mode.
>>> IIUC, the YARN UI only shows Flink clusters, not jobs. Instead, the jobs
>>> will be shown
>>> in Flink ui.
>>>
>>> Best,
>>> Weihua
>>>
>>>
>>> On Tue, Mar 28, 2023 at 11:16 PM Martijn Visser <
>>> martijnvisser@apache.org> wrote:
>>>
>>>> Hi,
>>>>
>>>> I don't see anything in the 1.10 release notes [1] that would indicate
>>>> that anything has changed between 1.9 and 1.10 that would indicate that
>>>> something has changed on this topic. It should work.
>>>>
>>>> Best regards,
>>>>
>>>> Martijn
>>>>
>>>> [1]
>>>> https://nightlies.apache.org/flink/flink-docs-release-1.10/release-notes/flink-1.10.html
>>>>
>>>> On Tue, Mar 28, 2023 at 4:40 PM Kiran Kumar Kathe <
>>>> kirankumarkathe123@gmail.com> wrote:
>>>>
>>>>> Actually we are using 1.10.1 in YARN cluster in production , so to
>>>>> make changes in cluster it is bit complicated and it need to check all the
>>>>> jobs from first and need to test everything , so is there any chance of
>>>>> running the with the flink 1.10.1 by doing any configuration changes , that
>>>>> make jobs visible in yarn Web UI.
>>>>>
>>>>> On Tue, 28 Mar 2023 at 19:59, Martijn Visser <ma...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> You can't mix and match different versions; they all need to be the
>>>>>> same version. Flink 1.9.3 is no longer supported by the Flink community, I
>>>>>> would recommend upgrading to a still supported version (which is currently
>>>>>> Flink 1.16 and Flink 1.17)
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Martijn
>>>>>>
>>>>>> On Tue, Mar 28, 2023 at 11:44 AM Kiran Kumar Kathe <
>>>>>> kirankumarkathe123@gmail.com> wrote:
>>>>>>
>>>>>>> When I submit jobs to flink using version 1.9.3 those jobs are
>>>>>>> visible in YARN resource manager Web UI , but when I use flink version
>>>>>>> 1.10.1 jobs running in flink are not visible in the Web UI , So in order
>>>>>>> to check where the issue is raising , I started replacing flink 1.10.1
>>>>>>> folders with flink 1.9.3 then , I get to know that lib folder can one jar
>>>>>>> file named flink_dist which is responsible , to make the jobs running in
>>>>>>> the flink visible in YARN resource manager. So my question here is Can I
>>>>>>> use the flink_dist jar of version 1.9.3 in place of flink_dist 1.10.1 , if
>>>>>>> not can anyone please give me clarity what configuration changes should i
>>>>>>> do in order to make jobs running visible in YARN resource manager Web UI
>>>>>>> with Flink version 1.10.1. [ I observed that while I am using flink 1.9.3
>>>>>>> it is starting a yarn-session and I can see logs related to that and when
>>>>>>> using flink 1.10.1 there are no logs related to yarn-session are visible.]
>>>>>>>
>>>>>>
Re: Jobs running in flink are not viable in YARN resource manager WEB UI
Posted by Weihua Hu <hu...@gmail.com>.
I see. So the screenshots are using version 1.9.3. What does version 1.10.1
show?
Best,
Weihua
On Wed, Mar 29, 2023 at 12:15 PM Kiran Kumar Kathe <
kirankumarkathe123@gmail.com> wrote:
> I am using yarn-cluster mode and command as ./flink run -m yarn-cluster
> /home/kirankathe/s3checkup/original-demo-helloworld-0.0.1-SNAPSHOT.jar
> and it is visible in the yarn UI like this as completed and the terminal
> looks as in the screenshots attached
>
> On Wed, 29 Mar 2023 at 09:31, Weihua Hu <hu...@gmail.com> wrote:
>
>> Hi,
>>
>>
>>> jobs are visible in YARN resource manager Web UI
>>
>> Which mode are you using to submit a job to YARN? Session mode or Perjob
>> mode.
>> IIUC, the YARN UI only shows Flink clusters, not jobs. Instead, the jobs
>> will be shown
>> in Flink ui.
>>
>> Best,
>> Weihua
>>
>>
>> On Tue, Mar 28, 2023 at 11:16 PM Martijn Visser <ma...@apache.org>
>> wrote:
>>
>>> Hi,
>>>
>>> I don't see anything in the 1.10 release notes [1] that would indicate
>>> that anything has changed between 1.9 and 1.10 that would indicate that
>>> something has changed on this topic. It should work.
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> [1]
>>> https://nightlies.apache.org/flink/flink-docs-release-1.10/release-notes/flink-1.10.html
>>>
>>> On Tue, Mar 28, 2023 at 4:40 PM Kiran Kumar Kathe <
>>> kirankumarkathe123@gmail.com> wrote:
>>>
>>>> Actually we are using 1.10.1 in YARN cluster in production , so to make
>>>> changes in cluster it is bit complicated and it need to check all the jobs
>>>> from first and need to test everything , so is there any chance of
>>>> running the with the flink 1.10.1 by doing any configuration changes , that
>>>> make jobs visible in yarn Web UI.
>>>>
>>>> On Tue, 28 Mar 2023 at 19:59, Martijn Visser <ma...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> You can't mix and match different versions; they all need to be the
>>>>> same version. Flink 1.9.3 is no longer supported by the Flink community, I
>>>>> would recommend upgrading to a still supported version (which is currently
>>>>> Flink 1.16 and Flink 1.17)
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Martijn
>>>>>
>>>>> On Tue, Mar 28, 2023 at 11:44 AM Kiran Kumar Kathe <
>>>>> kirankumarkathe123@gmail.com> wrote:
>>>>>
>>>>>> When I submit jobs to flink using version 1.9.3 those jobs are
>>>>>> visible in YARN resource manager Web UI , but when I use flink version
>>>>>> 1.10.1 jobs running in flink are not visible in the Web UI , So in order
>>>>>> to check where the issue is raising , I started replacing flink 1.10.1
>>>>>> folders with flink 1.9.3 then , I get to know that lib folder can one jar
>>>>>> file named flink_dist which is responsible , to make the jobs running in
>>>>>> the flink visible in YARN resource manager. So my question here is Can I
>>>>>> use the flink_dist jar of version 1.9.3 in place of flink_dist 1.10.1 , if
>>>>>> not can anyone please give me clarity what configuration changes should i
>>>>>> do in order to make jobs running visible in YARN resource manager Web UI
>>>>>> with Flink version 1.10.1. [ I observed that while I am using flink 1.9.3
>>>>>> it is starting a yarn-session and I can see logs related to that and when
>>>>>> using flink 1.10.1 there are no logs related to yarn-session are visible.]
>>>>>>
>>>>>
Re: Jobs running in flink are not viable in YARN resource manager WEB UI
Posted by Kiran Kumar Kathe <ki...@gmail.com>.
I am using yarn-cluster mode and command as ./flink run -m yarn-cluster
/home/kirankathe/s3checkup/original-demo-helloworld-0.0.1-SNAPSHOT.jar
and it is visible in the yarn UI like this as completed and the terminal
looks as in the screenshots attached
On Wed, 29 Mar 2023 at 09:31, Weihua Hu <hu...@gmail.com> wrote:
> Hi,
>
>
>> jobs are visible in YARN resource manager Web UI
>
> Which mode are you using to submit a job to YARN? Session mode or Perjob
> mode.
> IIUC, the YARN UI only shows Flink clusters, not jobs. Instead, the jobs
> will be shown
> in Flink ui.
>
> Best,
> Weihua
>
>
> On Tue, Mar 28, 2023 at 11:16 PM Martijn Visser <ma...@apache.org>
> wrote:
>
>> Hi,
>>
>> I don't see anything in the 1.10 release notes [1] that would indicate
>> that anything has changed between 1.9 and 1.10 that would indicate that
>> something has changed on this topic. It should work.
>>
>> Best regards,
>>
>> Martijn
>>
>> [1]
>> https://nightlies.apache.org/flink/flink-docs-release-1.10/release-notes/flink-1.10.html
>>
>> On Tue, Mar 28, 2023 at 4:40 PM Kiran Kumar Kathe <
>> kirankumarkathe123@gmail.com> wrote:
>>
>>> Actually we are using 1.10.1 in YARN cluster in production , so to make
>>> changes in cluster it is bit complicated and it need to check all the jobs
>>> from first and need to test everything , so is there any chance of
>>> running the with the flink 1.10.1 by doing any configuration changes , that
>>> make jobs visible in yarn Web UI.
>>>
>>> On Tue, 28 Mar 2023 at 19:59, Martijn Visser <ma...@apache.org>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> You can't mix and match different versions; they all need to be the
>>>> same version. Flink 1.9.3 is no longer supported by the Flink community, I
>>>> would recommend upgrading to a still supported version (which is currently
>>>> Flink 1.16 and Flink 1.17)
>>>>
>>>> Best regards,
>>>>
>>>> Martijn
>>>>
>>>> On Tue, Mar 28, 2023 at 11:44 AM Kiran Kumar Kathe <
>>>> kirankumarkathe123@gmail.com> wrote:
>>>>
>>>>> When I submit jobs to flink using version 1.9.3 those jobs are
>>>>> visible in YARN resource manager Web UI , but when I use flink version
>>>>> 1.10.1 jobs running in flink are not visible in the Web UI , So in order
>>>>> to check where the issue is raising , I started replacing flink 1.10.1
>>>>> folders with flink 1.9.3 then , I get to know that lib folder can one jar
>>>>> file named flink_dist which is responsible , to make the jobs running in
>>>>> the flink visible in YARN resource manager. So my question here is Can I
>>>>> use the flink_dist jar of version 1.9.3 in place of flink_dist 1.10.1 , if
>>>>> not can anyone please give me clarity what configuration changes should i
>>>>> do in order to make jobs running visible in YARN resource manager Web UI
>>>>> with Flink version 1.10.1. [ I observed that while I am using flink 1.9.3
>>>>> it is starting a yarn-session and I can see logs related to that and when
>>>>> using flink 1.10.1 there are no logs related to yarn-session are visible.]
>>>>>
>>>>
Re: Jobs running in flink are not viable in YARN resource manager WEB UI
Posted by Weihua Hu <hu...@gmail.com>.
Hi,
> jobs are visible in YARN resource manager Web UI
Which mode are you using to submit a job to YARN? Session mode or Perjob
mode.
IIUC, the YARN UI only shows Flink clusters, not jobs. Instead, the jobs
will be shown
in Flink ui.
Best,
Weihua
On Tue, Mar 28, 2023 at 11:16 PM Martijn Visser <ma...@apache.org>
wrote:
> Hi,
>
> I don't see anything in the 1.10 release notes [1] that would indicate
> that anything has changed between 1.9 and 1.10 that would indicate that
> something has changed on this topic. It should work.
>
> Best regards,
>
> Martijn
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.10/release-notes/flink-1.10.html
>
> On Tue, Mar 28, 2023 at 4:40 PM Kiran Kumar Kathe <
> kirankumarkathe123@gmail.com> wrote:
>
>> Actually we are using 1.10.1 in YARN cluster in production , so to make
>> changes in cluster it is bit complicated and it need to check all the jobs
>> from first and need to test everything , so is there any chance of
>> running the with the flink 1.10.1 by doing any configuration changes , that
>> make jobs visible in yarn Web UI.
>>
>> On Tue, 28 Mar 2023 at 19:59, Martijn Visser <ma...@apache.org>
>> wrote:
>>
>>> Hi,
>>>
>>> You can't mix and match different versions; they all need to be the same
>>> version. Flink 1.9.3 is no longer supported by the Flink community, I would
>>> recommend upgrading to a still supported version (which is currently Flink
>>> 1.16 and Flink 1.17)
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> On Tue, Mar 28, 2023 at 11:44 AM Kiran Kumar Kathe <
>>> kirankumarkathe123@gmail.com> wrote:
>>>
>>>> When I submit jobs to flink using version 1.9.3 those jobs are
>>>> visible in YARN resource manager Web UI , but when I use flink version
>>>> 1.10.1 jobs running in flink are not visible in the Web UI , So in order
>>>> to check where the issue is raising , I started replacing flink 1.10.1
>>>> folders with flink 1.9.3 then , I get to know that lib folder can one jar
>>>> file named flink_dist which is responsible , to make the jobs running in
>>>> the flink visible in YARN resource manager. So my question here is Can I
>>>> use the flink_dist jar of version 1.9.3 in place of flink_dist 1.10.1 , if
>>>> not can anyone please give me clarity what configuration changes should i
>>>> do in order to make jobs running visible in YARN resource manager Web UI
>>>> with Flink version 1.10.1. [ I observed that while I am using flink 1.9.3
>>>> it is starting a yarn-session and I can see logs related to that and when
>>>> using flink 1.10.1 there are no logs related to yarn-session are visible.]
>>>>
>>>
Re: Jobs running in flink are not viable in YARN resource manager WEB UI
Posted by Martijn Visser <ma...@apache.org>.
Hi,
I don't see anything in the 1.10 release notes [1] that would indicate that
anything has changed between 1.9 and 1.10 that would indicate that
something has changed on this topic. It should work.
Best regards,
Martijn
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.10/release-notes/flink-1.10.html
On Tue, Mar 28, 2023 at 4:40 PM Kiran Kumar Kathe <
kirankumarkathe123@gmail.com> wrote:
> Actually we are using 1.10.1 in YARN cluster in production , so to make
> changes in cluster it is bit complicated and it need to check all the jobs
> from first and need to test everything , so is there any chance of
> running the with the flink 1.10.1 by doing any configuration changes , that
> make jobs visible in yarn Web UI.
>
> On Tue, 28 Mar 2023 at 19:59, Martijn Visser <ma...@apache.org>
> wrote:
>
>> Hi,
>>
>> You can't mix and match different versions; they all need to be the same
>> version. Flink 1.9.3 is no longer supported by the Flink community, I would
>> recommend upgrading to a still supported version (which is currently Flink
>> 1.16 and Flink 1.17)
>>
>> Best regards,
>>
>> Martijn
>>
>> On Tue, Mar 28, 2023 at 11:44 AM Kiran Kumar Kathe <
>> kirankumarkathe123@gmail.com> wrote:
>>
>>> When I submit jobs to flink using version 1.9.3 those jobs are
>>> visible in YARN resource manager Web UI , but when I use flink version
>>> 1.10.1 jobs running in flink are not visible in the Web UI , So in order
>>> to check where the issue is raising , I started replacing flink 1.10.1
>>> folders with flink 1.9.3 then , I get to know that lib folder can one jar
>>> file named flink_dist which is responsible , to make the jobs running in
>>> the flink visible in YARN resource manager. So my question here is Can I
>>> use the flink_dist jar of version 1.9.3 in place of flink_dist 1.10.1 , if
>>> not can anyone please give me clarity what configuration changes should i
>>> do in order to make jobs running visible in YARN resource manager Web UI
>>> with Flink version 1.10.1. [ I observed that while I am using flink 1.9.3
>>> it is starting a yarn-session and I can see logs related to that and when
>>> using flink 1.10.1 there are no logs related to yarn-session are visible.]
>>>
>>
Re: Jobs running in flink are not viable in YARN resource manager WEB UI
Posted by Kiran Kumar Kathe <ki...@gmail.com>.
Actually we are using 1.10.1 in YARN cluster in production , so to make
changes in cluster it is bit complicated and it need to check all the jobs
from first and need to test everything , so is there any chance of
running the with the flink 1.10.1 by doing any configuration changes , that
make jobs visible in yarn Web UI.
On Tue, 28 Mar 2023 at 19:59, Martijn Visser <ma...@apache.org>
wrote:
> Hi,
>
> You can't mix and match different versions; they all need to be the same
> version. Flink 1.9.3 is no longer supported by the Flink community, I would
> recommend upgrading to a still supported version (which is currently Flink
> 1.16 and Flink 1.17)
>
> Best regards,
>
> Martijn
>
> On Tue, Mar 28, 2023 at 11:44 AM Kiran Kumar Kathe <
> kirankumarkathe123@gmail.com> wrote:
>
>> When I submit jobs to flink using version 1.9.3 those jobs are visible in
>> YARN resource manager Web UI , but when I use flink version 1.10.1 jobs
>> running in flink are not visible in the Web UI , So in order to check where
>> the issue is raising , I started replacing flink 1.10.1 folders with
>> flink 1.9.3 then , I get to know that lib folder can one jar file named
>> flink_dist which is responsible , to make the jobs running in the flink
>> visible in YARN resource manager. So my question here is Can I use the
>> flink_dist jar of version 1.9.3 in place of flink_dist 1.10.1 , if not can
>> anyone please give me clarity what configuration changes should i do in
>> order to make jobs running visible in YARN resource manager Web UI with
>> Flink version 1.10.1. [ I observed that while I am using flink 1.9.3 it is
>> starting a yarn-session and I can see logs related to that and when using
>> flink 1.10.1 there are no logs related to yarn-session are visible.]
>>
>
Re: Jobs running in flink are not viable in YARN resource manager WEB UI
Posted by Martijn Visser <ma...@apache.org>.
Hi,
You can't mix and match different versions; they all need to be the same
version. Flink 1.9.3 is no longer supported by the Flink community, I would
recommend upgrading to a still supported version (which is currently Flink
1.16 and Flink 1.17)
Best regards,
Martijn
On Tue, Mar 28, 2023 at 11:44 AM Kiran Kumar Kathe <
kirankumarkathe123@gmail.com> wrote:
> When I submit jobs to flink using version 1.9.3 those jobs are visible in
> YARN resource manager Web UI , but when I use flink version 1.10.1 jobs
> running in flink are not visible in the Web UI , So in order to check where
> the issue is raising , I started replacing flink 1.10.1 folders with
> flink 1.9.3 then , I get to know that lib folder can one jar file named
> flink_dist which is responsible , to make the jobs running in the flink
> visible in YARN resource manager. So my question here is Can I use the
> flink_dist jar of version 1.9.3 in place of flink_dist 1.10.1 , if not can
> anyone please give me clarity what configuration changes should i do in
> order to make jobs running visible in YARN resource manager Web UI with
> Flink version 1.10.1. [ I observed that while I am using flink 1.9.3 it is
> starting a yarn-session and I can see logs related to that and when using
> flink 1.10.1 there are no logs related to yarn-session are visible.]
>