You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Lian Jiang <ji...@gmail.com> on 2020/10/14 03:48:33 UTC

Stateful function and large state applications

Hi,

I am learning Stateful function and saw below:

"In addition to the Apache Flink processes, a full deployment requires
ZooKeeper <https://zookeeper.apache.org/> (for master failover
<https://ci.apache.org/projects/flink/flink-docs-stable/ops/jobmanager_high_availability.html>)
and bulk storage (S3, HDFS, NAS, GCS, Azure Blob Store, etc.) to store
Flink’s checkpoints
<https://ci.apache.org/projects/flink/flink-docs-master/concepts/stateful-stream-processing.html#checkpointing>.
In turn, the deployment requires no database, and Flink processes do not
require persistent volumes."

Does this mean stateful function does not support rocksdb (and incremental
checkpoint, local task recovery)? Will it be an issue for large state (e.g.
200GB) applications? Thanks for clarifying.


Thanks
Lian

Re: Stateful function and large state applications

Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.

Hi,

The StateFun runtime is built directly on top of Apache Flink, so RocksDB
as the state backend is supported as well as all the features for large
state such as checkpointing and local task recovery.

Cheers,
Gordon


On Wed, Oct 14, 2020 at 11:49 AM Lian Jiang <ji...@gmail.com> wrote:

> Hi,
>
> I am learning Stateful function and saw below:
>
> "In addition to the Apache Flink processes, a full deployment requires
> ZooKeeper <https://zookeeper.apache.org/> (for master failover
> <https://ci.apache.org/projects/flink/flink-docs-stable/ops/jobmanager_high_availability.html>)
> and bulk storage (S3, HDFS, NAS, GCS, Azure Blob Store, etc.) to store
> Flink’s checkpoints
> <https://ci.apache.org/projects/flink/flink-docs-master/concepts/stateful-stream-processing.html#checkpointing>.
> In turn, the deployment requires no database, and Flink processes do not
> require persistent volumes."
>
> Does this mean stateful function does not support rocksdb (and incremental
> checkpoint, local task recovery)? Will it be an issue for large state (e.g.
> 200GB) applications? Thanks for clarifying.
>
>
> Thanks
> Lian
>