You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by aljoscha <gi...@git.apache.org> on 2016/08/09 13:18:30 UTC
[GitHub] flink pull request #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint ...
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/2345
[FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
This was causing to many problems and is also incompatible with the
upcoming key-group/sharding changes.
R: @StephanEwen for review, should be fairly easy, though
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink rocksdb/remove-semi-async
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2345.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2345
----
commit 521f93c01bc1b99805d41eee5ab9f5279f52c4bd
Author: Aljoscha Krettek <al...@gmail.com>
Date: 2016-08-09T13:16:59Z
[FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
This was causing to many problems and is also incompatible with the
upcoming key-group/sharding changes.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2345
@wenlong88 I was talking about this effort to enable key-group sharding in Flink: https://issues.apache.org/jira/browse/FLINK-3755
With this is becomes necessary to checkpoint the keyed states in a way that operators can restore the state for the key shard for which they are responsible. The state backend therefore has to write the snapshot data such that keys for different key-groups are separated. If we keep the approach of simply checkpointing the whole RocksDB DB we don't get this property. In the fully-async checkpointing mode we iterate over the keys for each key group and write them separately from the keys of the other key groups. (logically speaking. they can still go into the same file but we need an index that tells us where the state for each key group resides.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2345
Jip, that's also good. \U0001f603
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2345
We are very much trying to avoid different db instances per operator. Each db instance will add quite a big new memory footprint. For higher max-parallelism, this will be a problem.
The current philosophy in Flink was to avoid such situations, where a user has to very carefully choose a parameter (like max parallelism) and where a wrong setting for the parameter would lead to an unstable system.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by gyfora <gi...@git.apache.org>.
Github user gyfora commented on the issue:
https://github.com/apache/flink/pull/2345
But I wonder what would happen in a scenario with a lot of states:
Semi async: short local copy time at every snapshot + very fast restore
Fully async: no copy time + very slow restore (puts sort data, recreate index etc)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2345
+1 for this
There seems to be an issue with the RocksDB backup engine, to we should probably discourage that mode even in current releases.
I would also remove the `HDFSCopyFromLocal` and `HDFSCopyToLocal` utils. We should not have dead code in the repository and we can always re-add them from the git history if needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by gyfora <gi...@git.apache.org>.
Github user gyfora commented on the issue:
https://github.com/apache/flink/pull/2345
But you are right it is probably more important to keep the latency down for the running programs, and for that the fully async seems to be strictly better
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by gyfora <gi...@git.apache.org>.
Github user gyfora commented on the issue:
https://github.com/apache/flink/pull/2345
Good thing about the way fully async checkpoints are restored though is that it is very trivial to insert some state adaptor code :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2345
@wenlong88 Can you explain this a bit more? How exactly are you creating the local checkpoint?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by wenlong88 <gi...@git.apache.org>.
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2345
@aljoscha I am curious at what problem and incompatibility between semi-aync snapshot and key-group, can you explain some background information?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2345
@gyfora You mean if the "full async" is slower than the "semi async"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint ...
Posted by aljoscha <gi...@git.apache.org>.
GitHub user aljoscha reopened a pull request:
https://github.com/apache/flink/pull/2345
[FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
This was causing to many problems and is also incompatible with the
upcoming key-group/sharding changes.
R: @StephanEwen for review, should be fairly easy, though
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink rocksdb/remove-semi-async
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2345.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2345
----
commit 521f93c01bc1b99805d41eee5ab9f5279f52c4bd
Author: Aljoscha Krettek <al...@gmail.com>
Date: 2016-08-09T13:16:59Z
[FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
This was causing to many problems and is also incompatible with the
upcoming key-group/sharding changes.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by uce <gi...@git.apache.org>.
Github user uce commented on the issue:
https://github.com/apache/flink/pull/2345
+1 to merge this.
On a related note: we might want to add information to the checkpoint statistics part of the web frontend saying whether the checkpoints were async or sync. Users might get confused when they see that the checkpoint times increased.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by wenlong88 <gi...@git.apache.org>.
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2345
@aljoscha we use rocksdb checkpoint machanism to do the semi-async checkpoint, which use hard link to make checkpoint, cost quite a little IO and time in synchronized phrase. This works well even when the state is large. and using the checkpoint dir to restore is also very fast, since no extra IO need, if the task manager didn't changed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2345
@StephanEwen That would probably be good, on a side note, time is calculated in the `CheckpointCoordinator` as `<time we received ack> - <time we started checkpoint>` right now.
This is somewhat confusing because it does not give the actual time it took to perform the checkpoint but the time since the whole global process was started at the checkpoint coordinator. Maybe the task has to send back some meta-data along with the actual checkpoint data.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by wenlong88 <gi...@git.apache.org>.
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2345
@StephanEwen http://rocksdb.org/blog/2609/use-checkpoints-for-efficient-snapshots/
since sst files is immutable once created, in when doing checkpoint, rocksdb creates hard link for all live sst files to a given directory and copy manifest to the directory, it cost little IO and the checkpoint dir can be used as data dir to restore
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint ...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/2345
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by gyfora <gi...@git.apache.org>.
Github user gyfora commented on the issue:
https://github.com/apache/flink/pull/2345
Some of the benefits we lose on restore. Especially for very large states this can be pretty serious.
Maybe this is required for the sharding to some extent but I don't see this as completely straightforward, in terms of which one is better in production.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by gyfora <gi...@git.apache.org>.
Github user gyfora commented on the issue:
https://github.com/apache/flink/pull/2345
Hi,
Isn't this way of checkpointing is much much slower then the semi async version?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2345
The "full async" takes more time but runs completely in the background, so performs better in most cases than "semi async".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by wenlong88 <gi...@git.apache.org>.
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2345
3q, @aljoscha
Maybe we can use different db instance for different key group. this approach can only work well when key group number is limited to a certain number which can not be too large.
Since the max number of key groups is limited by the max parallelism config, I think different db instances can work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2345
We can keep calculating the total time in the checkpoint coordinator. That way, it included message roundtrips.
The state handles (or whatever is the successor to them) should have the sync/async time components.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by gyfora <gi...@git.apache.org>.
Github user gyfora commented on the issue:
https://github.com/apache/flink/pull/2345
I can see the merits of both checkpointing approaches but Stephan is right in the sense that allowing semi-async snapshots with dynamic scaling would need a completely new implementation and would probably add considerable overhead.
I agree that we should probably remove it for the time being and maybe later we can reintroduce it if there is an efficient implementation with scaling in mind.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2345
Technically, `HDFSCopyFromLocal` and `HDFSCopyToLocal` are now unused. Should we remove them? They might be useful for some stuff in the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint ...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/2345
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by wenlong88 <gi...@git.apache.org>.
Github user wenlong88 commented on the issue:
https://github.com/apache/flink/pull/2345
@StephanEwen you are right. But in specific situation, we may need some temporary compromise do make the system work well, and then remove the compromised points latter as soon as possible .
I think both approaches have shortcomes. When the state is large such as millions of KVs per db, full async approach can do the full async backup, but will cost a lot of time to restore which may be intolerable while doing fail-over in production. So I think it is necessary to have both, and the full async can be the default option.
Considering that there is no really perfect solution yet, I think It is OK to remove the semi-async way right now to avoid blocking the job of key group but need to reintroduce a better solution latter soon if you agree that rocksdb is quite a good choice of statebackend in large state situations.
Regrading to the overhead of memory in different dbs. Rocksdb can share the same block cache for different db instance but I don't know how to reduce the cost of memtables which is also a problem existed in current solution that allowing to store different stats in a single db using column families since memtables of column families are also separated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2345: [FLINK-4340] Remove RocksDB Semi-Async Checkpoint Mode
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2345
How about having two times: The "synchronous" component and the "asynchronous" component of the time. Plus probably a total for those that don't want to sum them up themselves.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---