You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Peter Westermann <no...@genesys.com> on 2021/09/08 14:11:06 UTC
Duplicate copies of job in Flink UI/API
We recently upgraded from Flink 1.12.4 to 1.12.5 and are seeing some weird behavior after a change in jobmanager leadership: We’re seeing two copies of the same job, one of those is in SUSPENDED state and has a start time of zero. Here’s the output from the /jobs/overview endpoint:
{
"jobs": [{
"jid": "2db4ee6397151a1109d1ca05188a4cbb",
"name": "analytics-flink-v1",
"state": "RUNNING",
"start-time": 1631106146284,
"end-time": -1,
"duration": 2954642,
"last-modification": 1631106152322,
"tasks": {
"total": 112,
"created": 0,
"scheduled": 0,
"deploying": 0,
"running": 112,
"finished": 0,
"canceling": 0,
"canceled": 0,
"failed": 0,
"reconciling": 0
}
}, {
"jid": "2db4ee6397151a1109d1ca05188a4cbb",
"name": "analytics-flink-v1",
"state": "SUSPENDED",
"start-time": 0,
"end-time": -1,
"duration": 1631105900760,
"last-modification": 0,
"tasks": {
"total": 0,
"created": 0,
"scheduled": 0,
"deploying": 0,
"running": 0,
"finished": 0,
"canceling": 0,
"canceled": 0,
"failed": 0,
"reconciling": 0
}
}]
}
Has anyone seen this behavior before?
Thanks,
Peter
Re: Duplicate copies of job in Flink UI/API
Posted by Chesnay Schepler <ch...@apache.org>.
I'm afraid there's no real workaround.
If the information for completed jobs isn't important to you then
setting jobstore.expiration-time to a low value can reduce the impact,
or setting jobstore.max-capacity to 0 would prevent any completed job
from being displayed.
Beyond that I can't think of anything.
We will track the issue in this ticket:
https://issues.apache.org/jira/browse/FLINK-20195
I will also file a related ticket that the job archiving fails because,
again, we archive multiple jobs with the same job ID.
On 09/09/2021 15:15, Peter Westermann wrote:
>
> Thanks Chesnay. You are understanding this correctly. Your explanation
> makes sense to me.
>
> Is there anything we can do to prevent this? At least for us, most
> times a leader election happens, the leader doesn’t actually change
> because the jobmanager is still healthy.
>
> Thanks,
>
> Peter
>
>
> *From: *Chesnay Schepler <ch...@apache.org>
> *Date: *Thursday, September 9, 2021 at 9:11 AM
> *To: *Peter Westermann <no...@genesys.com>, Piotr Nowojski
> <pn...@apache.org>, user@flink.apache.org <us...@flink.apache.org>
> *Subject: *Re: Duplicate copies of job in Flink UI/API
>
> Just to double-check that I'm understanding things correctly:
>
> You have a job with HA, then Zookeeper breaks down, the job gets
> suspended, ZK comes back online, and the _same_ JobManager becomes the
> leader?
>
> If so, then I can explain why this happens and hopefully reproduce it.
>
> In short, when a job is suspended (or terminates in any other way)
> then information about the job is stored in a data-structure.
>
> This is used by the REST API (and thus, UI) to query completed jobs.
>
> For _running_ jobs we query the JobMaster (a component within the
> JobManager responsible for that job) instead.
>
> When listing all jobs we query all jobs from the data-structure for
> finished jobs, _and_ all JobMasters for running jobs. The core
> assumption here is that for a given ID only one of these can return
> something.
>
> So what ends up happening is that when the job is suspended it is
> written to the data-structure, and then another JobMaster is started
> for the same job, and when listing all jobs we can now end up asking
> for the same job from multiple sources.
>
> This is a somewhat unusual scenario because usually when a job is
> suspended another JobManager becomes the leader (where this wouldn't
> occur because the data-structure isn't shared across JobManagers).
>
> On 09/09/2021 13:37, Peter Westermann wrote:
>
> Hi Piotr,
>
> Jobmanager logs are attached to this email. The only thing that
> jumps out to me is this:
>
> 09/08/2021 09:02:26.240 -0400 ERROR
> org.apache.flink.runtime.history.FsJobArchivist Failed to archive job.
>
> java.io.IOException: File already
> exists:s3p://flink-s3-bucket/history/2db4ee6397151a1109d1ca05188a4cbb
>
> This happened days after the Flink update – and not just once.
> Across all our Flink clusters I’ve seen this 3 times. The cause
> for the jobmanager leadership loss in this case was a deployment
> of our zookeeper cluster that lead to a brief connection loss. The
> new leader election is expected.
>
> Thanks,
>
> Peter
>
> *From: *Piotr Nowojski <pn...@apache.org>
> <ma...@apache.org>
> *Date: *Thursday, September 9, 2021 at 12:39 AM
> *To: *Peter Westermann <no...@genesys.com>
> <ma...@genesys.com>
> *Cc: *user@flink.apache.org <ma...@flink.apache.org>
> <us...@flink.apache.org> <ma...@flink.apache.org>
> *Subject: *Re: Duplicate copies of job in Flink UI/API
>
> Hi Peter,
>
> Can you provide relevant JobManager logs? And can you write down
> what steps have you taken before the failure happened? Did this
> failure occur during upgrading Flink, or after the upgrade etc.
>
> Best,
>
> Piotrek
>
> śr., 8 wrz 2021 o 16:11 Peter Westermann
> <no.Westermann@genesys.com <ma...@genesys.com>>
> napisał(a):
>
> We recently upgraded from Flink 1.12.4 to 1.12.5 and are
> seeing some weird behavior after a change in jobmanager
> leadership: We’re seeing two copies of the same job, one of
> those is in SUSPENDED state and has a start time of zero.
> Here’s the output from the /jobs/overview endpoint:
>
> {
>
> "jobs": [{
>
> "jid": "2db4ee6397151a1109d1ca05188a4cbb",
>
> "name": "analytics-flink-v1",
>
> "state": "RUNNING",
>
> "start-time": 1631106146284,
>
> "end-time": -1,
>
> "duration": 2954642,
>
> "last-modification": 1631106152322,
>
> "tasks": {
>
> "total": 112,
>
> "created": 0,
>
> "scheduled": 0,
>
> "deploying": 0,
>
> "running": 112,
>
> "finished": 0,
>
> "canceling": 0,
>
> "canceled": 0,
>
> "failed": 0,
>
> "reconciling": 0
>
> }
>
> }, {
>
> "jid": "2db4ee6397151a1109d1ca05188a4cbb",
>
> "name": "analytics-flink-v1",
>
> "state": "SUSPENDED",
>
> "start-time": 0,
>
> "end-time": -1,
>
> "duration": 1631105900760,
>
> "last-modification": 0,
>
> "tasks": {
>
> "total": 0,
>
> "created": 0,
>
> "scheduled": 0,
>
> "deploying": 0,
>
> "running": 0,
>
> "finished": 0,
>
> "canceling": 0,
>
> "canceled": 0,
>
> "failed": 0,
>
> "reconciling": 0
>
> }
>
> }]
>
> }
>
> Has anyone seen this behavior before?
>
> Thanks,
>
> Peter
>
Re: Duplicate copies of job in Flink UI/API
Posted by Peter Westermann <no...@genesys.com>.
Thanks Chesnay. You are understanding this correctly. Your explanation makes sense to me.
Is there anything we can do to prevent this? At least for us, most times a leader election happens, the leader doesn’t actually change because the jobmanager is still healthy.
Thanks,
Peter
From: Chesnay Schepler <ch...@apache.org>
Date: Thursday, September 9, 2021 at 9:11 AM
To: Peter Westermann <no...@genesys.com>, Piotr Nowojski <pn...@apache.org>, user@flink.apache.org <us...@flink.apache.org>
Subject: Re: Duplicate copies of job in Flink UI/API
Just to double-check that I'm understanding things correctly:
You have a job with HA, then Zookeeper breaks down, the job gets suspended, ZK comes back online, and the _same_ JobManager becomes the leader?
If so, then I can explain why this happens and hopefully reproduce it.
In short, when a job is suspended (or terminates in any other way) then information about the job is stored in a data-structure.
This is used by the REST API (and thus, UI) to query completed jobs.
For _running_ jobs we query the JobMaster (a component within the JobManager responsible for that job) instead.
When listing all jobs we query all jobs from the data-structure for finished jobs, _and_ all JobMasters for running jobs. The core assumption here is that for a given ID only one of these can return something.
So what ends up happening is that when the job is suspended it is written to the data-structure, and then another JobMaster is started for the same job, and when listing all jobs we can now end up asking for the same job from multiple sources.
This is a somewhat unusual scenario because usually when a job is suspended another JobManager becomes the leader (where this wouldn't occur because the data-structure isn't shared across JobManagers).
On 09/09/2021 13:37, Peter Westermann wrote:
Hi Piotr,
Jobmanager logs are attached to this email. The only thing that jumps out to me is this:
09/08/2021 09:02:26.240 -0400 ERROR org.apache.flink.runtime.history.FsJobArchivist Failed to archive job.
java.io.IOException: File already exists:s3p://flink-s3-bucket/history/2db4ee6397151a1109d1ca05188a4cbb
This happened days after the Flink update – and not just once. Across all our Flink clusters I’ve seen this 3 times. The cause for the jobmanager leadership loss in this case was a deployment of our zookeeper cluster that lead to a brief connection loss. The new leader election is expected.
Thanks,
Peter
From: Piotr Nowojski <pn...@apache.org>
Date: Thursday, September 9, 2021 at 12:39 AM
To: Peter Westermann <no...@genesys.com>
Cc: user@flink.apache.org<ma...@flink.apache.org> <us...@flink.apache.org>
Subject: Re: Duplicate copies of job in Flink UI/API
Hi Peter,
Can you provide relevant JobManager logs? And can you write down what steps have you taken before the failure happened? Did this failure occur during upgrading Flink, or after the upgrade etc.
Best,
Piotrek
śr., 8 wrz 2021 o 16:11 Peter Westermann <no...@genesys.com>> napisał(a):
We recently upgraded from Flink 1.12.4 to 1.12.5 and are seeing some weird behavior after a change in jobmanager leadership: We’re seeing two copies of the same job, one of those is in SUSPENDED state and has a start time of zero. Here’s the output from the /jobs/overview endpoint:
{
"jobs": [{
"jid": "2db4ee6397151a1109d1ca05188a4cbb",
"name": "analytics-flink-v1",
"state": "RUNNING",
"start-time": 1631106146284,
"end-time": -1,
"duration": 2954642,
"last-modification": 1631106152322,
"tasks": {
"total": 112,
"created": 0,
"scheduled": 0,
"deploying": 0,
"running": 112,
"finished": 0,
"canceling": 0,
"canceled": 0,
"failed": 0,
"reconciling": 0
}
}, {
"jid": "2db4ee6397151a1109d1ca05188a4cbb",
"name": "analytics-flink-v1",
"state": "SUSPENDED",
"start-time": 0,
"end-time": -1,
"duration": 1631105900760,
"last-modification": 0,
"tasks": {
"total": 0,
"created": 0,
"scheduled": 0,
"deploying": 0,
"running": 0,
"finished": 0,
"canceling": 0,
"canceled": 0,
"failed": 0,
"reconciling": 0
}
}]
}
Has anyone seen this behavior before?
Thanks,
Peter
Re: Duplicate copies of job in Flink UI/API
Posted by Chesnay Schepler <ch...@apache.org>.
Just to double-check that I'm understanding things correctly:
You have a job with HA, then Zookeeper breaks down, the job gets
suspended, ZK comes back online, and the _same_ JobManager becomes the
leader?
If so, then I can explain why this happens and hopefully reproduce it.
In short, when a job is suspended (or terminates in any other way) then
information about the job is stored in a data-structure.
This is used by the REST API (and thus, UI) to query completed jobs.
For _running_ jobs we query the JobMaster (a component within the
JobManager responsible for that job) instead.
When listing all jobs we query all jobs from the data-structure for
finished jobs, _and_ all JobMasters for running jobs. The core
assumption here is that for a given ID only one of these can return
something.
So what ends up happening is that when the job is suspended it is
written to the data-structure, and then another JobMaster is started for
the same job, and when listing all jobs we can now end up asking for the
same job from multiple sources.
This is a somewhat unusual scenario because usually when a job is
suspended another JobManager becomes the leader (where this wouldn't
occur because the data-structure isn't shared across JobManagers).
On 09/09/2021 13:37, Peter Westermann wrote:
>
> Hi Piotr,
>
> Jobmanager logs are attached to this email. The only thing that jumps
> out to me is this:
>
> 09/08/2021 09:02:26.240 -0400 ERROR
> org.apache.flink.runtime.history.FsJobArchivist Failed to archive job.
>
> java.io.IOException: File already
> exists:s3p://flink-s3-bucket/history/2db4ee6397151a1109d1ca05188a4cbb
>
> This happened days after the Flink update – and not just once. Across
> all our Flink clusters I’ve seen this 3 times. The cause for the
> jobmanager leadership loss in this case was a deployment of our
> zookeeper cluster that lead to a brief connection loss. The new leader
> election is expected.
>
> Thanks,
>
> Peter
>
> *From: *Piotr Nowojski <pn...@apache.org>
> *Date: *Thursday, September 9, 2021 at 12:39 AM
> *To: *Peter Westermann <no...@genesys.com>
> *Cc: *user@flink.apache.org <us...@flink.apache.org>
> *Subject: *Re: Duplicate copies of job in Flink UI/API
>
> Hi Peter,
>
> Can you provide relevant JobManager logs? And can you write down what
> steps have you taken before the failure happened? Did this
> failure occur during upgrading Flink, or after the upgrade etc.
>
> Best,
>
> Piotrek
>
> śr., 8 wrz 2021 o 16:11 Peter Westermann <no.Westermann@genesys.com
> <ma...@genesys.com>> napisał(a):
>
> We recently upgraded from Flink 1.12.4 to 1.12.5 and are seeing
> some weird behavior after a change in jobmanager leadership: We’re
> seeing two copies of the same job, one of those is in SUSPENDED
> state and has a start time of zero. Here’s the output from the
> /jobs/overview endpoint:
>
> {
>
> "jobs": [{
>
> "jid": "2db4ee6397151a1109d1ca05188a4cbb",
>
> "name": "analytics-flink-v1",
>
> "state": "RUNNING",
>
> "start-time": 1631106146284,
>
> "end-time": -1,
>
> "duration": 2954642,
>
> "last-modification": 1631106152322,
>
> "tasks": {
>
> "total": 112,
>
> "created": 0,
>
> "scheduled": 0,
>
> "deploying": 0,
>
> "running": 112,
>
> "finished": 0,
>
> "canceling": 0,
>
> "canceled": 0,
>
> "failed": 0,
>
> "reconciling": 0
>
> }
>
> }, {
>
> "jid": "2db4ee6397151a1109d1ca05188a4cbb",
>
> "name": "analytics-flink-v1",
>
> "state": "SUSPENDED",
>
> "start-time": 0,
>
> "end-time": -1,
>
> "duration": 1631105900760,
>
> "last-modification": 0,
>
> "tasks": {
>
> "total": 0,
>
> "created": 0,
>
> "scheduled": 0,
>
> "deploying": 0,
>
> "running": 0,
>
> "finished": 0,
>
> "canceling": 0,
>
> "canceled": 0,
>
> "failed": 0,
>
> "reconciling": 0
>
> }
>
> }]
>
> }
>
> Has anyone seen this behavior before?
>
> Thanks,
>
> Peter
>
Re: Duplicate copies of job in Flink UI/API
Posted by Peter Westermann <no...@genesys.com>.
Hi Piotr,
Jobmanager logs are attached to this email. The only thing that jumps out to me is this:
09/08/2021 09:02:26.240 -0400 ERROR org.apache.flink.runtime.history.FsJobArchivist Failed to archive job.
java.io.IOException: File already exists:s3p://flink-s3-bucket/history/2db4ee6397151a1109d1ca05188a4cbb
This happened days after the Flink update – and not just once. Across all our Flink clusters I’ve seen this 3 times. The cause for the jobmanager leadership loss in this case was a deployment of our zookeeper cluster that lead to a brief connection loss. The new leader election is expected.
Thanks,
Peter
From: Piotr Nowojski <pn...@apache.org>
Date: Thursday, September 9, 2021 at 12:39 AM
To: Peter Westermann <no...@genesys.com>
Cc: user@flink.apache.org <us...@flink.apache.org>
Subject: Re: Duplicate copies of job in Flink UI/API
Hi Peter,
Can you provide relevant JobManager logs? And can you write down what steps have you taken before the failure happened? Did this failure occur during upgrading Flink, or after the upgrade etc.
Best,
Piotrek
śr., 8 wrz 2021 o 16:11 Peter Westermann <no...@genesys.com>> napisał(a):
We recently upgraded from Flink 1.12.4 to 1.12.5 and are seeing some weird behavior after a change in jobmanager leadership: We’re seeing two copies of the same job, one of those is in SUSPENDED state and has a start time of zero. Here’s the output from the /jobs/overview endpoint:
{
"jobs": [{
"jid": "2db4ee6397151a1109d1ca05188a4cbb",
"name": "analytics-flink-v1",
"state": "RUNNING",
"start-time": 1631106146284,
"end-time": -1,
"duration": 2954642,
"last-modification": 1631106152322,
"tasks": {
"total": 112,
"created": 0,
"scheduled": 0,
"deploying": 0,
"running": 112,
"finished": 0,
"canceling": 0,
"canceled": 0,
"failed": 0,
"reconciling": 0
}
}, {
"jid": "2db4ee6397151a1109d1ca05188a4cbb",
"name": "analytics-flink-v1",
"state": "SUSPENDED",
"start-time": 0,
"end-time": -1,
"duration": 1631105900760,
"last-modification": 0,
"tasks": {
"total": 0,
"created": 0,
"scheduled": 0,
"deploying": 0,
"running": 0,
"finished": 0,
"canceling": 0,
"canceled": 0,
"failed": 0,
"reconciling": 0
}
}]
}
Has anyone seen this behavior before?
Thanks,
Peter
Re: Duplicate copies of job in Flink UI/API
Posted by Piotr Nowojski <pn...@apache.org>.
Hi Peter,
Can you provide relevant JobManager logs? And can you write down what steps
have you taken before the failure happened? Did this failure occur during
upgrading Flink, or after the upgrade etc.
Best,
Piotrek
śr., 8 wrz 2021 o 16:11 Peter Westermann <no...@genesys.com>
napisał(a):
> We recently upgraded from Flink 1.12.4 to 1.12.5 and are seeing some weird
> behavior after a change in jobmanager leadership: We’re seeing two copies
> of the same job, one of those is in SUSPENDED state and has a start time of
> zero. Here’s the output from the /jobs/overview endpoint:
>
> {
>
> "jobs": [{
>
> "jid": "2db4ee6397151a1109d1ca05188a4cbb",
>
> "name": "analytics-flink-v1",
>
> "state": "RUNNING",
>
> "start-time": 1631106146284,
>
> "end-time": -1,
>
> "duration": 2954642,
>
> "last-modification": 1631106152322,
>
> "tasks": {
>
> "total": 112,
>
> "created": 0,
>
> "scheduled": 0,
>
> "deploying": 0,
>
> "running": 112,
>
> "finished": 0,
>
> "canceling": 0,
>
> "canceled": 0,
>
> "failed": 0,
>
> "reconciling": 0
>
> }
>
> }, {
>
> "jid": "2db4ee6397151a1109d1ca05188a4cbb",
>
> "name": "analytics-flink-v1",
>
> "state": "SUSPENDED",
>
> "start-time": 0,
>
> "end-time": -1,
>
> "duration": 1631105900760,
>
> "last-modification": 0,
>
> "tasks": {
>
> "total": 0,
>
> "created": 0,
>
> "scheduled": 0,
>
> "deploying": 0,
>
> "running": 0,
>
> "finished": 0,
>
> "canceling": 0,
>
> "canceled": 0,
>
> "failed": 0,
>
> "reconciling": 0
>
> }
>
> }]
>
> }
>
>
>
> Has anyone seen this behavior before?
>
>
>
> Thanks,
>
> Peter
>