You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Marchant, Hayden " <ha...@citi.com> on 2017/08/31 07:53:46 UTC
Using local FS for checkpoint
Whether I use RocksDB or FS State backends, if my requirements are to have fault-tolerance and ability to recover with 'at-least once' semantics for my Flink job, is there still a valid case for using a backing local FS for storing states? i.e. If a Flink Node is invalidated, I would have thought that the only way it could recover (by re-starting the task on different node), is if the state is stored in a shared file system such as HDFS, S3 etc....
I am asking since I want to know if HDFS is a must have for my deployment.
Thanks,
Hayden
RE: Using local FS for checkpoint
Posted by prashantnayak <pr...@intellifylearning.com>.
We ran into issues using EFS (which under the covers is a NFS like
filesystem)... details are in this post
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/External-checkpoints-not-getting-cleaned-up-discarded-potentially-causing-high-load-tp14073p14106.html
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
RE: Using local FS for checkpoint
Posted by "Marchant, Hayden " <ha...@citi.com>.
I didn’t think about NFS. That would save me the hassle of installing HDFS cluster just for that, especially if my organization already has an NFS ‘handy’.
Thanks
Hayden
From: Tony Wei [mailto:tony19920430@gmail.com]
Sent: Thursday, August 31, 2017 12:12 PM
To: Marchant, Hayden [ICG-IT]
Cc: user@flink.apache.org
Subject: Re: Using local FS for checkpoint
Hi Marchant,
HDFS is not a must for storing checkpoints. S3 or NFS are all acceptable, as long as it is accessible from job manager and task manager.
For AWS S3 configuration, you can refer to this page (https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__ci.apache.org_projects_flink_flink-2Ddocs-2Drelease-2D1.3_setup_aws.html&d=DwMFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=g-5xYRH8L3aCnCNTROw5LrsB5gbTayWjXSm6Nil9x0c&m=FqFmOvFkeIVmEku6VDuGfbYpEZLcbN7UUGFUei8TrgA&s=Em489MhXME4sEtU03lj8groEt92mJWMIeAvcx7Hi7is&e=>).
Best,
Tony Wei
2017-08-31 15:53 GMT+08:00 Marchant, Hayden <ha...@citi.com>>:
Whether I use RocksDB or FS State backends, if my requirements are to have fault-tolerance and ability to recover with 'at-least once' semantics for my Flink job, is there still a valid case for using a backing local FS for storing states? i.e. If a Flink Node is invalidated, I would have thought that the only way it could recover (by re-starting the task on different node), is if the state is stored in a shared file system such as HDFS, S3 etc....
I am asking since I want to know if HDFS is a must have for my deployment.
Thanks,
Hayden
Re: Using local FS for checkpoint
Posted by Tony Wei <to...@gmail.com>.
Hi Marchant,
HDFS is not a must for storing checkpoints. S3 or NFS are all acceptable,
as long as it is accessible from job manager and task manager.
For AWS S3 configuration, you can refer to this page (
https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html).
Best,
Tony Wei
2017-08-31 15:53 GMT+08:00 Marchant, Hayden <ha...@citi.com>:
> Whether I use RocksDB or FS State backends, if my requirements are to have
> fault-tolerance and ability to recover with 'at-least once' semantics for
> my Flink job, is there still a valid case for using a backing local FS for
> storing states? i.e. If a Flink Node is invalidated, I would have thought
> that the only way it could recover (by re-starting the task on different
> node), is if the state is stored in a shared file system such as HDFS, S3
> etc....
>
> I am asking since I want to know if HDFS is a must have for my deployment.
>
> Thanks,
>
> Hayden
>
>
>