You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by amenreet sodhi <am...@gmail.com> on 2023/07/11 07:19:43 UTC
Local State Storage
Hi Team,
I wanted to confirm, the local state which TM stores in the directory
either we provide through config or default i.e. /tmp folder, does it clear
itself from time to time or the size just keeps on increasing?
Thanks
Regards
Amenreet Singh Sodhi
Re: Local State Storage
Posted by Shammon FY <zj...@gmail.com>.
Hi amenreet,
I think there are two ways to clean up state data in the flink job
automatically:
1. State TTL. You can configure the ttl [1] for state according to your
requirements, and flink job will clean the data when they are out of date.
For flink SQL jobs you can set a global ttl for all operators and set each
operator ttl like [2]
2. If you use rocksdb statebacked in your job, flink will delete the sst
files if they are not referenced by checkpoint any more.
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/state/#state-time-to-live-ttl
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-280%3A+Introduce+EXPLAIN+PLAN_ADVICE+to+provide+SQL+advice
Best,
Shammon FY
On Tue, Jul 11, 2023 at 3:20 PM amenreet sodhi <am...@gmail.com> wrote:
> Hi Team,
>
> I wanted to confirm, the local state which TM stores in the directory
> either we provide through config or default i.e. /tmp folder, does it clear
> itself from time to time or the size just keeps on increasing?
>
> Thanks
> Regards
> Amenreet Singh Sodhi
>
>