You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by RKandoji <rk...@gmail.com> on 2020/01/07 22:36:44 UTC
How to verify if checkpoints are asynchronous or sync
Hi,
I'm using Flink 1.9, BlinkPlanner and rocksDB for backend with
checkpointing enabled.
I understand that checkpointing is async by default for RocksDB. But I'm
curious if there is way to verify if the checkpoints are happening
asynchronously or synchronously.
Please let me know.
Thanks,
RKandoji
Re: How to verify if checkpoints are asynchronous or sync
Posted by RKandoji <rk...@gmail.com>.
I evaluated performance by looking at the number of input records processed
over 10min and 30mins duration.
Thanks,
R
On Wed, Jan 8, 2020 at 2:21 AM Congxian Qiu <qc...@gmail.com> wrote:
> If you want to figure out the performance problem, maybe async-profile[1]
> can be helpful
> [1] https://github.com/jvm-profiling-tools/async-profiler
> Best,
> Congxian
>
>
> William C <wi...@uubeta.com> 于2020年1月8日周三 上午11:37写道：
>
>> Hallo
>>
>> on 2020/1/8 11:31, RKandoji wrote:
>> > I'm running my job on a EC2 instance with 32 cores and according to the
>> > documentation I tried to use as many task slots the number of cores,
>> > numOfTaskSlots=32 and parallelism=32. But I noticed that
>> the performance
>> > is slightly degrading when I'm using 32 task slots. Performance seems
>> > better at 26 task slots than >26 task slots. So I was trying to
>> > understand if additional CPU cores are being utilized by checkpointing
>> > or any other async (or background operations, in the process I was
>> > trying to verify if the checkpointing is async.
>>
>> How did you evaluate the performance?
>> It's may due to busy IO or thread competition or something else similiar.
>> You'd better dig into more details via logging.
>>
>> Regards.
>>
>
Re: How to verify if checkpoints are asynchronous or sync
Posted by Congxian Qiu <qc...@gmail.com>.
If you want to figure out the performance problem, maybe async-profile[1]
can be helpful
[1] https://github.com/jvm-profiling-tools/async-profiler
Best,
Congxian
William C <wi...@uubeta.com> 于2020年1月8日周三 上午11:37写道：
> Hallo
>
> on 2020/1/8 11:31, RKandoji wrote:
> > I'm running my job on a EC2 instance with 32 cores and according to the
> > documentation I tried to use as many task slots the number of cores,
> > numOfTaskSlots=32 and parallelism=32. But I noticed that the performance
> > is slightly degrading when I'm using 32 task slots. Performance seems
> > better at 26 task slots than >26 task slots. So I was trying to
> > understand if additional CPU cores are being utilized by checkpointing
> > or any other async (or background operations, in the process I was
> > trying to verify if the checkpointing is async.
>
> How did you evaluate the performance?
> It's may due to busy IO or thread competition or something else similiar.
> You'd better dig into more details via logging.
>
> Regards.
>
Re: How to verify if checkpoints are asynchronous or sync
Posted by William C <wi...@uubeta.com>.
Hallo
on 2020/1/8 11:31, RKandoji wrote:
> I'm running my job on a EC2 instance with 32 cores and according to the
> documentation I tried to use as many task slots the number of cores,
> numOfTaskSlots=32 and parallelism=32. But I noticed that the performance
> is slightly degrading when I'm using 32 task slots. Performance seems
> better at 26 task slots than >26 task slots. So I was trying to
> understand if additional CPU cores are being utilized by checkpointing
> or any other async (or background operations, in the process I was
> trying to verify if the checkpointing is async.
How did you evaluate the performance?
It's may due to busy IO or thread competition or something else similiar.
You'd better dig into more details via logging.
Regards.
Re: How to verify if checkpoints are asynchronous or sync
Posted by RKandoji <rk...@gmail.com>.
Sorry for not providing the context earlier.
I'm running my job on a EC2 instance with 32 cores and according to the
documentation I tried to use as many task slots the number of cores,
numOfTaskSlots=32 and parallelism=32. But I noticed that the performance is
slightly degrading when I'm using 32 task slots. Performance seems better
at 26 task slots than >26 task slots. So I was trying to understand if
additional CPU cores are being utilized by checkpointing or any other async
(or background operations, in the process I was trying to verify if
the checkpointing is async.
Thanks,
RKandoji
On Tue, Jan 7, 2020 at 10:16 PM Congxian Qiu <qc...@gmail.com> wrote:
> Hi
>
> RocksDB supports Incremental and full snapshot, both are async, do you
> want to verify whether it's incremental or full snapshot? but I don't know
> an easy to get this information currently
>
> Best,
> Congxian
>
>
> Zhijiang <wa...@aliyun.com> 于2020年1月8日周三 上午10:56写道：
>
>> The log way is simple for tracing and you can also grep some keywords to
>> find your requirement messages to avoid skimming through the whole large
>> logs.
>> I am not quite sure what's your specific motivation for doing this.
>> Besides the log way, you can also monitor the thread stack for confirming
>> whether it is happening, but maybe it is not very convenient.
>> Another possible way is via the checkpoint metrics which would record the
>> sync/async duration time, maybe it can also satisfy your requirements.
>>
>> Best,
>> Zhijiang
>>
>> ------------------------------------------------------------------
>> From:RKandoji <rk...@gmail.com>
>> Send Time:2020 Jan. 8 (Wed.) 10:23
>> To:William C <wi...@uubeta.com>
>> Cc:user <us...@flink.apache.org>
>> Subject:Re: How to verify if checkpoints are asynchronous or sync
>>
>> Thanks for the reply.
>> I will check and enable debug logs specifically for the class that
>> contains this log.
>> But in general logs are already too huge and I'm trying to suppress some
>> of them, so wondering if there is any other way?
>>
>> Thanks,
>> RKandoji
>>
>>
>> On Tue, Jan 7, 2020 at 7:50 PM William C <wi...@uubeta.com> wrote:
>> Can you enable debug log to check with that?
>>
>> regards.
>>
>> on 2020/1/8 6:36, RKandoji wrote:
>> > But I'm curious if there is way to verify if the checkpoints are
>> > happening asynchronously or synchronously.
>> >
>>
>>
>>
Re: How to verify if checkpoints are asynchronous or sync
Posted by Congxian Qiu <qc...@gmail.com>.
Hi
RocksDB supports Incremental and full snapshot, both are async, do you want
to verify whether it's incremental or full snapshot? but I don't know an
easy to get this information currently
Best,
Congxian
Zhijiang <wa...@aliyun.com> 于2020年1月8日周三 上午10:56写道：
> The log way is simple for tracing and you can also grep some keywords to
> find your requirement messages to avoid skimming through the whole large
> logs.
> I am not quite sure what's your specific motivation for doing this.
> Besides the log way, you can also monitor the thread stack for confirming
> whether it is happening, but maybe it is not very convenient.
> Another possible way is via the checkpoint metrics which would record the
> sync/async duration time, maybe it can also satisfy your requirements.
>
> Best,
> Zhijiang
>
> ------------------------------------------------------------------
> From:RKandoji <rk...@gmail.com>
> Send Time:2020 Jan. 8 (Wed.) 10:23
> To:William C <wi...@uubeta.com>
> Cc:user <us...@flink.apache.org>
> Subject:Re: How to verify if checkpoints are asynchronous or sync
>
> Thanks for the reply.
> I will check and enable debug logs specifically for the class that
> contains this log.
> But in general logs are already too huge and I'm trying to suppress some
> of them, so wondering if there is any other way?
>
> Thanks,
> RKandoji
>
>
> On Tue, Jan 7, 2020 at 7:50 PM William C <wi...@uubeta.com> wrote:
> Can you enable debug log to check with that?
>
> regards.
>
> on 2020/1/8 6:36, RKandoji wrote:
> > But I'm curious if there is way to verify if the checkpoints are
> > happening asynchronously or synchronously.
> >
>
>
>
Re: How to verify if checkpoints are asynchronous or sync
Posted by Zhijiang <wa...@aliyun.com>.
The log way is simple for tracing and you can also grep some keywords to find your requirement messages to avoid skimming through the whole large logs.
I am not quite sure what's your specific motivation for doing this. Besides the log way, you can also monitor the thread stack for confirming whether it is happening, but maybe it is not very convenient.
Another possible way is via the checkpoint metrics which would record the sync/async duration time, maybe it can also satisfy your requirements.
Best,
Zhijiang
------------------------------------------------------------------
From:RKandoji <rk...@gmail.com>
Send Time:2020 Jan. 8 (Wed.) 10:23
To:William C <wi...@uubeta.com>
Cc:user <us...@flink.apache.org>
Subject:Re: How to verify if checkpoints are asynchronous or sync
Thanks for the reply.
I will check and enable debug logs specifically for the class that contains this log.
But in general logs are already too huge and I'm trying to suppress some of them, so wondering if there is any other way?
Thanks,
RKandoji
On Tue, Jan 7, 2020 at 7:50 PM William C <wi...@uubeta.com> wrote:
Can you enable debug log to check with that?
regards.
on 2020/1/8 6:36, RKandoji wrote:
> But I'm curious if there is way to verify if the checkpoints are
> happening asynchronously or synchronously.
>
Re: How to verify if checkpoints are asynchronous or sync
Posted by RKandoji <rk...@gmail.com>.
Thanks for the reply.
I will check and enable debug logs specifically for the class that contains
this log.
But in general logs are already too huge and I'm trying to suppress some of
them, so wondering if there is any other way?
Thanks,
RKandoji
On Tue, Jan 7, 2020 at 7:50 PM William C <wi...@uubeta.com> wrote:
> Can you enable debug log to check with that?
>
> regards.
>
> on 2020/1/8 6:36, RKandoji wrote:
> > But I'm curious if there is way to verify if the checkpoints are
> > happening asynchronously or synchronously.
> >
>
Re: How to verify if checkpoints are asynchronous or sync
Posted by William C <wi...@uubeta.com>.
Can you enable debug log to check with that?
regards.
on 2020/1/8 6:36, RKandoji wrote:
> But I'm curious if there is way to verify if the checkpoints are
> happening asynchronously or synchronously.
>