Multi-Datacentre Failover

[Marcus Horsley-Rai <ma...@gmail.com>]

Hi,

I am working on a project that has to be resilient and highly available,
and would like to be able to deploy a Flink/Kafka setup as active/active
across two data centres (Live/Hot backup in a sense).
I came across an old forum post asking how failover could be performed if
one datacenter goes down, and the answer was to try and automate a solution
involving savepoints.

In my case, there would be independent Kafka clusters in each data centre,
but input data would be mirrored to both.

The documentation mentions that Flink saves the Kafka consumer offsets as
part of the checkpoint/savepoint, and resumes from those offsets on a
restore.
If I has a solution involving syncing checkpoints between DCs, what
would be the behaviour given the offsets of Flink in DC1 will not marry up
exactly with those in DC2?
e.g. will it just pick up from latest?

Many thanks in advance,

Marcus

Re: Multi-Datacentre Failover

[Marta Paes Moreira <ma...@ververica.com>]

Hey Marcus!

This mailing list is mostly for community-bound announcements and
discussions, not user support. Please use *user@flink.apache.org
<us...@flink.apache.org>* for that.

Someone will be able to help you there, usually within a few hours!

Marta

On Tue, Apr 7, 2020 at 2:44 PM Marcus Horsley-Rai <ma...@gmail.com> wrote:

> Hi,
>
> I am working on a project that has to be resilient and highly available,
> and would like to be able to deploy a Flink/Kafka setup as active/active
> across two data centres (Live/Hot backup in a sense).
> I came across an old forum post asking how failover could be performed if
> one datacenter goes down, and the answer was to try and automate a solution
> involving savepoints.
>
> In my case, there would be independent Kafka clusters in each data centre,
> but input data would be mirrored to both.
>
> The documentation mentions that Flink saves the Kafka consumer offsets as
> part of the checkpoint/savepoint, and resumes from those offsets on a
> restore.
> If I has a solution involving syncing checkpoints between DCs, what
> would be the behaviour given the offsets of Flink in DC1 will not marry up
> exactly with those in DC2?
> e.g. will it just pick up from latest?
>
> Many thanks in advance,
>
> Marcus
>