quesiont related to store location of save point when using file:///, instead of s3 and hdfs

[Sendoh <un...@gmail.com>]

Hi Flink users,

When using `file:///mysavepoint` instead of `hdsf://mysavepoint  for
ENVCONFIG.setString("state.checkpoints.dir", stateBackend);

savepoint will be transfered from task manager to job manager (same as using
memory as state backend), and stored in the disk of job manager?

Savepoint seems to be necessary to be stored in the place where job manager
can find, and then it can start the job again. It looks like it can work
without s3 or hdfs , is that true? ( if this works, that means we can use
EBS)

Best,

Sendoh



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/quesiont-related-to-store-location-of-save-point-when-using-file-instead-of-s3-and-hdfs-tp12435.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: quesiont related to store location of save point when using file:///, instead of s3 and hdfs

[Fabian Hueske <fh...@gmail.com>]

Hi,

if you use the MemoryStateBackend, the state is shipped from the TMs to the
JM. The JM holds the state in its own memory (i.e., on the JVM heap).
This means that the MemoryStateBackend is not reliable by design and rather
meant for testing and running Flink in your IDE.

If you want to persist the state in a distributed filesystem, you should
use the FSStateBackend. Similar to the MemoryStateBackend, the
FSStateBackend stores the data on the TM's heap, but on a checkpoint the
state is written into a configurable FS directory.

Best, Fabian

2017-03-28 18:00 GMT+02:00 Sendoh <un...@gmail.com>:

> Hi Flink users,
>
> When using `file:///mysavepoint` instead of `hdsf://mysavepoint  for
> ENVCONFIG.setString("state.checkpoints.dir", stateBackend);
>
> savepoint will be transfered from task manager to job manager (same as
> using
> memory as state backend), and stored in the disk of job manager?
>
> Savepoint seems to be necessary to be stored in the place where job manager
> can find, and then it can start the job again. It looks like it can work
> without s3 or hdfs , is that true? ( if this works, that means we can use
> EBS)
>
> Best,
>
> Sendoh
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/quesiont-
> related-to-store-location-of-save-point-when-using-file-
> instead-of-s3-and-hdfs-tp12435.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>