You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Dan Hill <qu...@gmail.com> on 2021/03/01 20:34:56 UTC
Debugging long Flink checkpoint durations
Hi. Are there good ways to debug long Flink checkpoint durations?
I'm running a backfill job that runs ~10 days of data and then starts
checkpointing failing. Since I only see the last 10 checkpoints in the
jobmaster UI, I don't see when it starts.
I looked through the text logs and didn't see much.
I assume:
1) I have something misconfigured that is causing old state is sticking
around.
2) I don't have enough resources.
Re: Debugging long Flink checkpoint durations
Posted by Dan Hill <qu...@gmail.com>.
The checkpoint was only acknowledged shortly after it was started.
On Thu, Mar 4, 2021 at 12:38 PM Dan Hill <qu...@gmail.com> wrote:
> I dove deeper into it and made a little more progress (by giving more
> resources).
>
> Here is a screenshot of one bottleneck:
> https://drive.google.com/file/d/1CIatEuIJwmKjBE9__RihVlxSilchtKS1/view
>
> My job isn't making any progress. It's checkpointing and failing. The
> taskmaster text logs are empty during the checkpoint. It's not clear if
> the checkpoint is making any progress.
>
> https://drive.google.com/file/d/1slLO6PJVhXfoAN5OrSqsE9G7kvHPXJnl/view?usp=sharing
>
> I spent some time changing the memory parameters but it's unclear if I'm
> making forward progress.
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_tm.html
>
>
>
>
> On Tue, Mar 2, 2021 at 3:45 PM Dan Hill <qu...@gmail.com> wrote:
>
>> Thanks! Yes, I've looked at these. My job is facing backpressure
>> starting at an early join step. I'm unclear if more time is fine for the
>> backfill or if I need more resources.
>>
>> On Tue, Mar 2, 2021 at 12:50 AM Yun Gao <yu...@aliyun.com> wrote:
>>
>>> Hi Dan,
>>>
>>> I think you could see the detail of the checkpoints via the checkpoint
>>> UI[1]. Also, if you see in the
>>> pending checkpoints some tasks do not take snapshot, you might have a
>>> look whether this task
>>> is backpressuring the previous tasks [2].
>>>
>>> Best,
>>> Yun
>>>
>>>
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/monitoring/checkpoint_monitoring.html
>>> [2]
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/monitoring/back_pressure.html
>>>
>>> ------------------------------------------------------------------
>>> Sender:Dan Hill<qu...@gmail.com>
>>> Date:2021/03/02 04:34:56
>>> Recipient:user<us...@flink.apache.org>
>>> Theme:Debugging long Flink checkpoint durations
>>>
>>> Hi. Are there good ways to debug long Flink checkpoint durations?
>>>
>>> I'm running a backfill job that runs ~10 days of data and then starts
>>> checkpointing failing. Since I only see the last 10 checkpoints in the
>>> jobmaster UI, I don't see when it starts.
>>>
>>> I looked through the text logs and didn't see much.
>>>
>>> I assume:
>>> 1) I have something misconfigured that is causing old state is sticking
>>> around.
>>> 2) I don't have enough resources.
>>>
>>>
>>>
Re: Debugging long Flink checkpoint durations
Posted by Dan Hill <qu...@gmail.com>.
I dove deeper into it and made a little more progress (by giving more
resources).
Here is a screenshot of one bottleneck:
https://drive.google.com/file/d/1CIatEuIJwmKjBE9__RihVlxSilchtKS1/view
My job isn't making any progress. It's checkpointing and failing. The
taskmaster text logs are empty during the checkpoint. It's not clear if
the checkpoint is making any progress.
https://drive.google.com/file/d/1slLO6PJVhXfoAN5OrSqsE9G7kvHPXJnl/view?usp=sharing
I spent some time changing the memory parameters but it's unclear if I'm
making forward progress.
https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/memory/mem_setup_tm.html
On Tue, Mar 2, 2021 at 3:45 PM Dan Hill <qu...@gmail.com> wrote:
> Thanks! Yes, I've looked at these. My job is facing backpressure
> starting at an early join step. I'm unclear if more time is fine for the
> backfill or if I need more resources.
>
> On Tue, Mar 2, 2021 at 12:50 AM Yun Gao <yu...@aliyun.com> wrote:
>
>> Hi Dan,
>>
>> I think you could see the detail of the checkpoints via the checkpoint
>> UI[1]. Also, if you see in the
>> pending checkpoints some tasks do not take snapshot, you might have a
>> look whether this task
>> is backpressuring the previous tasks [2].
>>
>> Best,
>> Yun
>>
>>
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/monitoring/checkpoint_monitoring.html
>> [2]
>> https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/monitoring/back_pressure.html
>>
>> ------------------------------------------------------------------
>> Sender:Dan Hill<qu...@gmail.com>
>> Date:2021/03/02 04:34:56
>> Recipient:user<us...@flink.apache.org>
>> Theme:Debugging long Flink checkpoint durations
>>
>> Hi. Are there good ways to debug long Flink checkpoint durations?
>>
>> I'm running a backfill job that runs ~10 days of data and then starts
>> checkpointing failing. Since I only see the last 10 checkpoints in the
>> jobmaster UI, I don't see when it starts.
>>
>> I looked through the text logs and didn't see much.
>>
>> I assume:
>> 1) I have something misconfigured that is causing old state is sticking
>> around.
>> 2) I don't have enough resources.
>>
>>
>>
Re: Debugging long Flink checkpoint durations
Posted by Dan Hill <qu...@gmail.com>.
Thanks! Yes, I've looked at these. My job is facing backpressure
starting at an early join step. I'm unclear if more time is fine for the
backfill or if I need more resources.
On Tue, Mar 2, 2021 at 12:50 AM Yun Gao <yu...@aliyun.com> wrote:
> Hi Dan,
>
> I think you could see the detail of the checkpoints via the checkpoint
> UI[1]. Also, if you see in the
> pending checkpoints some tasks do not take snapshot, you might have a
> look whether this task
> is backpressuring the previous tasks [2].
>
> Best,
> Yun
>
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/monitoring/checkpoint_monitoring.html
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/monitoring/back_pressure.html
>
> ------------------------------------------------------------------
> Sender:Dan Hill<qu...@gmail.com>
> Date:2021/03/02 04:34:56
> Recipient:user<us...@flink.apache.org>
> Theme:Debugging long Flink checkpoint durations
>
> Hi. Are there good ways to debug long Flink checkpoint durations?
>
> I'm running a backfill job that runs ~10 days of data and then starts
> checkpointing failing. Since I only see the last 10 checkpoints in the
> jobmaster UI, I don't see when it starts.
>
> I looked through the text logs and didn't see much.
>
> I assume:
> 1) I have something misconfigured that is causing old state is sticking
> around.
> 2) I don't have enough resources.
>
>
>
Re: Debugging long Flink checkpoint durations
Posted by Yun Gao <yu...@aliyun.com>.
Hi Dan,
I think you could see the detail of the checkpoints via the checkpoint UI[1]. Also, if you see in the
pending checkpoints some tasks do not take snapshot, you might have a look whether this task
is backpressuring the previous tasks [2].
Best,
Yun
[1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/monitoring/checkpoint_monitoring.html
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.12/ops/monitoring/back_pressure.html
------------------------------------------------------------------
Sender:Dan Hill<qu...@gmail.com>
Date:2021/03/02 04:34:56
Recipient:user<us...@flink.apache.org>
Theme:Debugging long Flink checkpoint durations
Hi. Are there good ways to debug long Flink checkpoint durations?
I'm running a backfill job that runs ~10 days of data and then starts checkpointing failing. Since I only see the last 10 checkpoints in the jobmaster UI, I don't see when it starts.
I looked through the text logs and didn't see much.
I assume:
1) I have something misconfigured that is causing old state is sticking around.
2) I don't have enough resources.