Flink support for multiple data centers

[Olga Luganska <tr...@hotmail.com>]

Hello,

Does Flink support multiple data center implementation and failover procedures in case one of the data centers goes down?

Another question I have is about data encryption. If  application state which needs to be checkpointed contains data elements which are considered to be a personally identifiable information, or maybe some credit card information, do you provide any encryption mechanisms to make sure that this data will be secured?

Thank you very much,
Olga

Re: Flink support for multiple data centers

[Andrey Zagrebin <an...@data-artisans.com>]

Hi Olga,

At the moment Flink does not have any embedded support for multi region deployment or failover.
You can try to automate it, for example, using savepoints [1] and backing them up in multiple data centres.
This way you could restore failed job in other regions from the latest savepoint.
Of course, job input/output should be also available in multiple data centres.

Flink does not have any embedded encryption mechanism for state.
The state value is just de-serialisable byte array for Flink.
For example, you can implement custom TypeSerializer [2] or wrap existing one with encryption.

[1] https://ci.apache.org/projects/flink/flink-docs-master/ops/state/savepoints.html
[2] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/state/custom_serialization.html

Best,
Andrey

> On 2 Oct 2018, at 02:57, Olga Luganska <tr...@hotmail.com> wrote:
> 
> Hello,
> 
> Does Flink support multiple data center implementation and failover procedures in case one of the data centers goes down?
> 
> Another question I have is about data encryption. If  application state which needs to be checkpointed contains data elements which are considered to be a personally identifiable information, or maybe some credit card information, do you provide any encryption mechanisms to make sure that this data will be secured?
> 
> Thank you very much,
> Olga