[DISCUSSION] Database state backend

[Gyula Fóra <gy...@apache.org>]

Hello everyone!

I would like to start a discussion regarding the future of the Database
state backend for the streaming API. The main question is whether we want
to keep this as a flink-contrib moduel and continue development as part of
Flink or should we move it to an outside library.

Just as a quick refresher, the DB state backend is designed to store
streaming state in JDBC supporting databases (the connector in the flink
repo supports MySQL). There is an adapter interface that can be implemented
to use different databases other than MySQL.

Advantages:

   - Can store huge states (tested with more than a TB of state with MySQL)
   - Fast, incremental checkpoints by default
   - In-memory cache for hot keys
   - Supports rescaling easily (restarting the job with different
   parallelism would still work as expected)
   - Supports database sharding with custom data partitioner
   - Exposes an easily accessible state view to the outside (will contain
   uncheckpointed state)

Disadvantages:

   - Lookup/Update might be much slower than local state backend (RocksDB
   for instance)
   - Incremental checkpoints are not very well supported with the current
   runtime so state cleanup/compaction does not happen automatically
   - Restore speed depends on database indexing as it involves range
   deletes based on timestamps (this might get slow on very large states)
   - The caching layer is not memory aware, this should be moved to flink
   managed memory

We developed (and contributed) the DB state backend to be able to handle
large state sizes, but have moved to RocksDB for much better read/write
performance. I have a pending PR with some improvements to the MySQL
adapter but the question is whether we should keep this as part of Flink
contrib or move it to an external library.

I am personally undecided whether there are strong use cases that would
favour this backend over RocksDB.

Any opinions?

Cheers,
Gyula

Re: [DISCUSSION] Database state backend

[Gyula Fóra <gy...@gmail.com>]

Hey,

Yes, I agree that it is probably best to remove this backend from Flink and
make it an external library instead.

I will prepare a JIRA and PR.

Gyula

Aljoscha Krettek <al...@apache.org> ezt írta (időpont: 2016. márc. 16.,
Sze, 10:35):

> Hi,
> if you yourself (Gyula) don’t want to maintain it anymore in the Flink
> codebase I would vote to move it to an external repository. If you are not
> using it anymore I’m afraid no one will really work on it.
>
> On more thing. When using the DB state backend savepoints don’t work.
> Cleanup/compaction mess with the stored state and a checkpoint/savepoint of
> the DB state backend is therefore not a self-contained unit.
>
> Cheers,
> Aljoscha
> > On 14 Mar 2016, at 12:30, Ufuk Celebi <uc...@apache.org> wrote:
> >
> > On Mon, Mar 14, 2016 at 11:20 AM, Gyula Fóra <gy...@apache.org> wrote:
> >> We developed (and contributed) the DB state backend to be able to handle
> >> large state sizes, but have moved to RocksDB for much better read/write
> >> performance. I have a pending PR with some improvements to the MySQL
> >> adapter but the question is whether we should keep this as part of Flink
> >> contrib or move it to an external library.
> >>
> >> I am personally undecided whether there are strong use cases that would
> >> favour this backend over RocksDB.
> >
> > Hey Gyula,
> >
> > thanks for starting this discussion and the great summary of Pros and
> > Cons. I fully agree that it boils down to whether there are strong use
> > cases that would favour the DB backend over RocksDB.
> >
> > Personally, I am in favour of moving the DB state backend out of core
> Flink:
> >
> > - I think that the DB state backend was a very valuable contribution,
> > but as you've already outlined, RocksDB seems to work better for large
> > out of core state, which was the prime motivation for the DB state
> > backend.
> >
> > - Furthermore, my experience from chatting with you while you were
> > evaluating the DB backend was that there are many parameters to tune
> > for this to work realiably (at least with MySQL). I'm wondering how
> > many users will have the same motivation to actually look into all of
> > these parameters. In my experience, RocksDB provides a better out of
> > the box experience, which is in line with Flink's over all philosophy.
> >
> > It would be possible to just keep the code around in the contrib
> > module, but if we don't have the resources to maintain it (including
> > more documentation and tests), I don't see the point.
> >
> > Since you are the main person driving this, the question is how much
> > time you can/want to invest into this. If you feel like that you
> > personally would really like to keep it in Flink and develop it
> > further here, I would certainly be in favour of keeping it. But at the
> > moment, I don't see this being the case.
> >
> > – Ufuk
>
>

Re: [DISCUSSION] Database state backend

[Aljoscha Krettek <al...@apache.org>]

Hi,
if you yourself (Gyula) don’t want to maintain it anymore in the Flink codebase I would vote to move it to an external repository. If you are not using it anymore I’m afraid no one will really work on it. 

On more thing. When using the DB state backend savepoints don’t work. Cleanup/compaction mess with the stored state and a checkpoint/savepoint of the DB state backend is therefore not a self-contained unit.

Cheers,
Aljoscha 
> On 14 Mar 2016, at 12:30, Ufuk Celebi <uc...@apache.org> wrote:
> 
> On Mon, Mar 14, 2016 at 11:20 AM, Gyula Fóra <gy...@apache.org> wrote:
>> We developed (and contributed) the DB state backend to be able to handle
>> large state sizes, but have moved to RocksDB for much better read/write
>> performance. I have a pending PR with some improvements to the MySQL
>> adapter but the question is whether we should keep this as part of Flink
>> contrib or move it to an external library.
>> 
>> I am personally undecided whether there are strong use cases that would
>> favour this backend over RocksDB.
> 
> Hey Gyula,
> 
> thanks for starting this discussion and the great summary of Pros and
> Cons. I fully agree that it boils down to whether there are strong use
> cases that would favour the DB backend over RocksDB.
> 
> Personally, I am in favour of moving the DB state backend out of core Flink:
> 
> - I think that the DB state backend was a very valuable contribution,
> but as you've already outlined, RocksDB seems to work better for large
> out of core state, which was the prime motivation for the DB state
> backend.
> 
> - Furthermore, my experience from chatting with you while you were
> evaluating the DB backend was that there are many parameters to tune
> for this to work realiably (at least with MySQL). I'm wondering how
> many users will have the same motivation to actually look into all of
> these parameters. In my experience, RocksDB provides a better out of
> the box experience, which is in line with Flink's over all philosophy.
> 
> It would be possible to just keep the code around in the contrib
> module, but if we don't have the resources to maintain it (including
> more documentation and tests), I don't see the point.
> 
> Since you are the main person driving this, the question is how much
> time you can/want to invest into this. If you feel like that you
> personally would really like to keep it in Flink and develop it
> further here, I would certainly be in favour of keeping it. But at the
> moment, I don't see this being the case.
> 
> – Ufuk

Re: [DISCUSSION] Database state backend

[Ufuk Celebi <uc...@apache.org>]

On Mon, Mar 14, 2016 at 11:20 AM, Gyula Fóra <gy...@apache.org> wrote:
> We developed (and contributed) the DB state backend to be able to handle
> large state sizes, but have moved to RocksDB for much better read/write
> performance. I have a pending PR with some improvements to the MySQL
> adapter but the question is whether we should keep this as part of Flink
> contrib or move it to an external library.
>
> I am personally undecided whether there are strong use cases that would
> favour this backend over RocksDB.

Hey Gyula,

thanks for starting this discussion and the great summary of Pros and
Cons. I fully agree that it boils down to whether there are strong use
cases that would favour the DB backend over RocksDB.

Personally, I am in favour of moving the DB state backend out of core Flink:

- I think that the DB state backend was a very valuable contribution,
but as you've already outlined, RocksDB seems to work better for large
out of core state, which was the prime motivation for the DB state
backend.

- Furthermore, my experience from chatting with you while you were
evaluating the DB backend was that there are many parameters to tune
for this to work realiably (at least with MySQL). I'm wondering how
many users will have the same motivation to actually look into all of
these parameters. In my experience, RocksDB provides a better out of
the box experience, which is in line with Flink's over all philosophy.

It would be possible to just keep the code around in the contrib
module, but if we don't have the resources to maintain it (including
more documentation and tests), I don't see the point.

Since you are the main person driving this, the question is how much
time you can/want to invest into this. If you feel like that you
personally would really like to keep it in Flink and develop it
further here, I would certainly be in favour of keeping it. But at the
moment, I don't see this being the case.

– Ufuk