You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Vishal Santoshi <vi...@gmail.com> on 2021/03/29 20:08:20 UTC
SP with Drain and Cancel hangs after take a SP
Is this a known issue. We do a stop + savepoint with drain. I see no back
pressure on our operators. It essentially takes a SP and then the SInk (
StreamingFileSink to S3 ) just stays in the RUNNING state.
Without drain i stop + savepoint works fine. I would imagine drain is
important ( flush the buffers etc ) but why this hang ( I did it 3 times
and waited 15 minutes each time ).
Regards.
Re: SP with Drain and Cancel hangs after take a SP
Posted by Vishal Santoshi <vi...@gmail.com>.
Great, thanks!
On Tue, Mar 30, 2021 at 11:00 AM Till Rohrmann <tr...@apache.org> wrote:
> This is a good idea. I will add it to the section here [1].
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/deployment/cli.html#terminating-a-job
>
> Cheers,
> Till
>
> On Tue, Mar 30, 2021 at 2:46 PM Vishal Santoshi <vi...@gmail.com>
> wrote:
>
>> Got it. Is it possible to add this very important note to the
>> documentation. Our case is the former as in this is an infinite pipeline
>> and we were establishing the CiCD release process when non breaking
>> changes ( DAG compatible changes are made ) on a running pipe.
>>
>> Regards
>>
>> On Tue, Mar 30, 2021 at 8:14 AM Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>>> Hi Vishal,
>>>
>>> The difference between stop-with-savepoint and
>>> stop-with-savepoint-with-drain is that the latter emits a max watermark
>>> before taking the snapshot. The idea is to trigger all pending timers and
>>> flush the content of some buffering operations like windowing.
>>> Semantically, you should use the first option if you want to stop the job
>>> and resume it at a later point in time. Stop-with-savepoint-with-drain
>>> should only be used if you want to terminate your job and don't intend to
>>> resume it because the max watermark destroys the correctness of results
>>> which are generated after the job is resumed.
>>>
>>> For the concrete problem at hand it is difficult to say why it does not
>>> stop. It would be helpful if you could provide us with the debug logs of
>>> such a run. I am also pulling Arvid who works on Flink's connector
>>> ecosystem.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Mon, Mar 29, 2021 at 11:08 PM Vishal Santoshi <
>>> vishal.santoshi@gmail.com> wrote:
>>>
>>>> More interested whether a StreamingFileSink without a drain
>>>> negatively affects it's exactly-once semantics , given that I state on SP
>>>> would have the offsets from kafka + the valid lengths of the part files at
>>>> SP. To be honest not sure whether the flushed buffers on sink are included
>>>> in the length, or this is not an issue with StreamingFileSink. If it is the
>>>> former then I would assume we should be documented and then have to look
>>>> why this hang happens.
>>>>
>>>> On Mon, Mar 29, 2021 at 4:08 PM Vishal Santoshi <
>>>> vishal.santoshi@gmail.com> wrote:
>>>>
>>>>> Is this a known issue. We do a stop + savepoint with drain. I see no
>>>>> back pressure on our operators. It essentially takes a SP and then the SInk
>>>>> ( StreamingFileSink to S3 ) just stays in the RUNNING state.
>>>>>
>>>>> Without drain i stop + savepoint works fine. I would imagine drain is
>>>>> important ( flush the buffers etc ) but why this hang ( I did it 3 times
>>>>> and waited 15 minutes each time ).
>>>>>
>>>>> Regards.
>>>>>
>>>>
Re: SP with Drain and Cancel hangs after take a SP
Posted by Till Rohrmann <tr...@apache.org>.
This is a good idea. I will add it to the section here [1].
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/deployment/cli.html#terminating-a-job
Cheers,
Till
On Tue, Mar 30, 2021 at 2:46 PM Vishal Santoshi <vi...@gmail.com>
wrote:
> Got it. Is it possible to add this very important note to the
> documentation. Our case is the former as in this is an infinite pipeline
> and we were establishing the CiCD release process when non breaking
> changes ( DAG compatible changes are made ) on a running pipe.
>
> Regards
>
> On Tue, Mar 30, 2021 at 8:14 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Hi Vishal,
>>
>> The difference between stop-with-savepoint and
>> stop-with-savepoint-with-drain is that the latter emits a max watermark
>> before taking the snapshot. The idea is to trigger all pending timers and
>> flush the content of some buffering operations like windowing.
>> Semantically, you should use the first option if you want to stop the job
>> and resume it at a later point in time. Stop-with-savepoint-with-drain
>> should only be used if you want to terminate your job and don't intend to
>> resume it because the max watermark destroys the correctness of results
>> which are generated after the job is resumed.
>>
>> For the concrete problem at hand it is difficult to say why it does not
>> stop. It would be helpful if you could provide us with the debug logs of
>> such a run. I am also pulling Arvid who works on Flink's connector
>> ecosystem.
>>
>> Cheers,
>> Till
>>
>> On Mon, Mar 29, 2021 at 11:08 PM Vishal Santoshi <
>> vishal.santoshi@gmail.com> wrote:
>>
>>> More interested whether a StreamingFileSink without a drain
>>> negatively affects it's exactly-once semantics , given that I state on SP
>>> would have the offsets from kafka + the valid lengths of the part files at
>>> SP. To be honest not sure whether the flushed buffers on sink are included
>>> in the length, or this is not an issue with StreamingFileSink. If it is the
>>> former then I would assume we should be documented and then have to look
>>> why this hang happens.
>>>
>>> On Mon, Mar 29, 2021 at 4:08 PM Vishal Santoshi <
>>> vishal.santoshi@gmail.com> wrote:
>>>
>>>> Is this a known issue. We do a stop + savepoint with drain. I see no
>>>> back pressure on our operators. It essentially takes a SP and then the SInk
>>>> ( StreamingFileSink to S3 ) just stays in the RUNNING state.
>>>>
>>>> Without drain i stop + savepoint works fine. I would imagine drain is
>>>> important ( flush the buffers etc ) but why this hang ( I did it 3 times
>>>> and waited 15 minutes each time ).
>>>>
>>>> Regards.
>>>>
>>>
Re: SP with Drain and Cancel hangs after take a SP
Posted by Vishal Santoshi <vi...@gmail.com>.
Got it. Is it possible to add this very important note to the
documentation. Our case is the former as in this is an infinite pipeline
and we were establishing the CiCD release process when non breaking
changes ( DAG compatible changes are made ) on a running pipe.
Regards
On Tue, Mar 30, 2021 at 8:14 AM Till Rohrmann <tr...@apache.org> wrote:
> Hi Vishal,
>
> The difference between stop-with-savepoint and
> stop-with-savepoint-with-drain is that the latter emits a max watermark
> before taking the snapshot. The idea is to trigger all pending timers and
> flush the content of some buffering operations like windowing.
> Semantically, you should use the first option if you want to stop the job
> and resume it at a later point in time. Stop-with-savepoint-with-drain
> should only be used if you want to terminate your job and don't intend to
> resume it because the max watermark destroys the correctness of results
> which are generated after the job is resumed.
>
> For the concrete problem at hand it is difficult to say why it does not
> stop. It would be helpful if you could provide us with the debug logs of
> such a run. I am also pulling Arvid who works on Flink's connector
> ecosystem.
>
> Cheers,
> Till
>
> On Mon, Mar 29, 2021 at 11:08 PM Vishal Santoshi <
> vishal.santoshi@gmail.com> wrote:
>
>> More interested whether a StreamingFileSink without a drain
>> negatively affects it's exactly-once semantics , given that I state on SP
>> would have the offsets from kafka + the valid lengths of the part files at
>> SP. To be honest not sure whether the flushed buffers on sink are included
>> in the length, or this is not an issue with StreamingFileSink. If it is the
>> former then I would assume we should be documented and then have to look
>> why this hang happens.
>>
>> On Mon, Mar 29, 2021 at 4:08 PM Vishal Santoshi <
>> vishal.santoshi@gmail.com> wrote:
>>
>>> Is this a known issue. We do a stop + savepoint with drain. I see no
>>> back pressure on our operators. It essentially takes a SP and then the SInk
>>> ( StreamingFileSink to S3 ) just stays in the RUNNING state.
>>>
>>> Without drain i stop + savepoint works fine. I would imagine drain is
>>> important ( flush the buffers etc ) but why this hang ( I did it 3 times
>>> and waited 15 minutes each time ).
>>>
>>> Regards.
>>>
>>
Re: SP with Drain and Cancel hangs after take a SP
Posted by Till Rohrmann <tr...@apache.org>.
Hi Vishal,
The difference between stop-with-savepoint and
stop-with-savepoint-with-drain is that the latter emits a max watermark
before taking the snapshot. The idea is to trigger all pending timers and
flush the content of some buffering operations like windowing.
Semantically, you should use the first option if you want to stop the job
and resume it at a later point in time. Stop-with-savepoint-with-drain
should only be used if you want to terminate your job and don't intend to
resume it because the max watermark destroys the correctness of results
which are generated after the job is resumed.
For the concrete problem at hand it is difficult to say why it does not
stop. It would be helpful if you could provide us with the debug logs of
such a run. I am also pulling Arvid who works on Flink's connector
ecosystem.
Cheers,
Till
On Mon, Mar 29, 2021 at 11:08 PM Vishal Santoshi <vi...@gmail.com>
wrote:
> More interested whether a StreamingFileSink without a drain
> negatively affects it's exactly-once semantics , given that I state on SP
> would have the offsets from kafka + the valid lengths of the part files at
> SP. To be honest not sure whether the flushed buffers on sink are included
> in the length, or this is not an issue with StreamingFileSink. If it is the
> former then I would assume we should be documented and then have to look
> why this hang happens.
>
> On Mon, Mar 29, 2021 at 4:08 PM Vishal Santoshi <vi...@gmail.com>
> wrote:
>
>> Is this a known issue. We do a stop + savepoint with drain. I see no back
>> pressure on our operators. It essentially takes a SP and then the SInk (
>> StreamingFileSink to S3 ) just stays in the RUNNING state.
>>
>> Without drain i stop + savepoint works fine. I would imagine drain is
>> important ( flush the buffers etc ) but why this hang ( I did it 3 times
>> and waited 15 minutes each time ).
>>
>> Regards.
>>
>
Re: SP with Drain and Cancel hangs after take a SP
Posted by Vishal Santoshi <vi...@gmail.com>.
More interested whether a StreamingFileSink without a drain
negatively affects it's exactly-once semantics , given that I state on SP
would have the offsets from kafka + the valid lengths of the part files at
SP. To be honest not sure whether the flushed buffers on sink are included
in the length, or this is not an issue with StreamingFileSink. If it is the
former then I would assume we should be documented and then have to look
why this hang happens.
On Mon, Mar 29, 2021 at 4:08 PM Vishal Santoshi <vi...@gmail.com>
wrote:
> Is this a known issue. We do a stop + savepoint with drain. I see no back
> pressure on our operators. It essentially takes a SP and then the SInk (
> StreamingFileSink to S3 ) just stays in the RUNNING state.
>
> Without drain i stop + savepoint works fine. I would imagine drain is
> important ( flush the buffers etc ) but why this hang ( I did it 3 times
> and waited 15 minutes each time ).
>
> Regards.
>