You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Dominik Wosiński <wo...@gmail.com> on 2018/06/21 09:01:04 UTC
JobManager restarts all jobs
Hey,
I have some kind of issue, possibly connected with configuration. When I
have started JobManager in HA mode *zookeeper**. *Everything works fine,
jobs are executed etc.
But when I Cancel any job or the job fails, after the reelection of the
leader, the new JobManager restarts all the jobs
*even those that were failed and cancelled. *
This causes failure of the JobManager since I am using Blob Server on HDFS
and if the job is cancelled or fails its blob data is immediately removed,
the JobManager tries to access BlobData that does not exist and throws no
such file exception.
The log looks like below :
Re: JobManager restarts all jobs
Posted by Till Rohrmann <tr...@apache.org>.
Hi Dominik,
could you share with us the cluster entrypoint log files. I guess this
problem is related to FLINK-9575 [1]. I think we should not delete the blob
files if the we could not delete the JobGraph from the submitted job graph
store.
[1] https://issues.apache.org/jira/browse/FLINK-9575.
Cheers,
Till
On Fri, Jun 22, 2018 at 9:57 AM Fabian Hueske <fh...@gmail.com> wrote:
> Hi Dominik,
>
> Which version are you running?
> Till (in CC) is most familiar with the job recovery and might be able to
> help.
>
> Best,
> Fabian
>
> 2018-06-21 11:01 GMT+02:00 Dominik Wosiński <wo...@gmail.com>:
>
>> Hey,
>> I have some kind of issue, possibly connected with configuration. When I
>> have started JobManager in HA mode *zookeeper**. *Everything works fine,
>> jobs are executed etc.
>>
>> But when I Cancel any job or the job fails, after the reelection of the
>> leader, the new JobManager restarts all the jobs
>> *even those that were failed and cancelled. *
>> This causes failure of the JobManager since I am using Blob Server on
>> HDFS and if the job is cancelled or fails its blob data is immediately
>> removed, the JobManager tries to access BlobData that does not exist and
>> throws no such file exception.
>>
>> The log looks like below :
>>
>>
>
Re: JobManager restarts all jobs
Posted by Fabian Hueske <fh...@gmail.com>.
Hi Dominik,
Which version are you running?
Till (in CC) is most familiar with the job recovery and might be able to
help.
Best,
Fabian
2018-06-21 11:01 GMT+02:00 Dominik Wosiński <wo...@gmail.com>:
> Hey,
> I have some kind of issue, possibly connected with configuration. When I
> have started JobManager in HA mode *zookeeper**. *Everything works fine,
> jobs are executed etc.
>
> But when I Cancel any job or the job fails, after the reelection of the
> leader, the new JobManager restarts all the jobs
> *even those that were failed and cancelled. *
> This causes failure of the JobManager since I am using Blob Server on HDFS
> and if the job is cancelled or fails its blob data is immediately removed,
> the JobManager tries to access BlobData that does not exist and throws no
> such file exception.
>
> The log looks like below :
>
>