Paimon table retention

[Kirill Lyashko <ki...@gmail.com>]

Hi all,

I have a Paimon table that is continuously populated by a streaming job
based on incoming data.
I need to evict records from the table once they are no longer used—where
"not used" can be defined by a SQL query.

From what I’ve observed, running such a query in a separate batch job
requires suspending the streaming job. This is because multiple writers to
the same Paimon table can cause conflicts, potentially resulting in job
failures.

Additionally, I haven't found a way to periodically schedule such a cleanup
query within the streaming application itself.

What would be the best way to handle this use case?