You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by uce <gi...@git.apache.org> on 2017/02/17 13:55:50 UTC

[GitHub] flink pull request #3346: Selfcontained

GitHub user uce opened a pull request:

    https://github.com/apache/flink/pull/3346

    Selfcontained

    This is based on #3345 and only the last two commits are relevant here. I've separated the test changes (last commit) and the main changes (2nd last commit) for better reviewability. I would squash them before merging this. The change is looks more involved than it actually is. It is mostly routing new information with the checkpointing barriers, which touches a lot of places.
    
    The main change is to add `CheckpointOptions` to the triggered checkpoint messages (coordinator to barrier injecting tasks) and barriers (flowing inline with the data):
    
    ```java
    public class CheckpointOptions {
    	
    	// Type of checkpoint
    	// => FULL_CHECKPOINT
    	// => SAVEPOINT
    	@NonNull
    	CheckpointType getCheckpointType();
    
    	// Custom target location. This is a String, because for future
    	// backends it can be a logical location like a DB table.
    	@Nullable
    	String getTargetLocation();
    
    }
    ```
    
    This class would be the place to define more options for performing the checkpoints (for example for incremental checkpoints). @StephanEwen was involved with the design of incremental checkpoints and could probably comment best whether this is inline with the design for that.
    
    These options are forwarded via the `StreamTask` to the `StreamOperator`s and `Snapshotable` backends. The `AbstractStreamOperator` checks the options and either
    i) forwards the shared per operator `CheckpointStreamFactory` (as of #3312), or
    ii) creates a custom savepoint stream factory (one per savepoint).
    
    For this, the state backends provide the following new method:
    
    ```java
    CheckpointStreamFactory createSavepointStreamFactory(JobID, String, String);
    ```
    
    The `MemoryStateBackend` returns the regular stream factory and the `FsStateBackend` returns a `FsSavepointStreamFactory`, which writes all checkpoint streams to a single directory (instead of the regular sub folders per checkpoint).
    
    We end up with the following directory layout for savepoints:
    
    ```
    +---------------------------+
    | :root_savepoint_directory | (custom per savepoint or configured default via `state.savepoints.dir`)
    +---------------------------+
      | +---------------------------------------+
      +-| savepoint-:jobId(0, 6)-:random_suffix | (one directory per savepoint)
        +---------------------------------------+
          |
          +- _metadata (one per savepoint)
          +- :uuid (one data file per StreamTask)
          +- ...
          +- :uuid
    ```
    
    I decided to include a prefix of the job ID to the savepoint directory, because I think that this could be helpful to map savepoints to jobs, which is a manual task. 
    
    It's important to make sure that this is inline with upcoming changes for incremental checkpoints (discussion on mailing list) and FLINK-5820.
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/uce/flink selfcontained

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/3346.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3346
    
----
commit 64e1f8892597dd26489774fe494fd7b211d789a9
Author: Ufuk Celebi <uc...@apache.org>
Date:   2017-02-15T16:52:40Z

    [FLINK-5763] [checkpoints] Acknowledge with explicit ID and CheckpointMetrics
    
    Instead of acknowledging checkpoints with the CheckpointMetaData make
    the acknowledgement explicit by ID and CheckpointMetrics. The rest is
    not needed.

commit 0b8fa02a150ae6d5891e04d1fce6de748a283aaf
Author: Ufuk Celebi <uc...@apache.org>
Date:   2017-02-15T17:16:44Z

    [FLINK-5763] [checkpoints] Move CheckpointMetrics out of CheckpointMetaData

commit d023159d0fef1da24373d7880e11d0a36afbd7ef
Author: Ufuk Celebi <uc...@apache.org>
Date:   2017-02-16T15:52:32Z

    [FLINK-5763] [checkpoints] Add isSavepoint() to CheckpointProperties

commit 2496068371bf1aac9e2ce6223002c1d9a043930f
Author: Ufuk Celebi <uc...@apache.org>
Date:   2017-02-16T16:56:23Z

    [FLINK-5763] [checkpoints] Add CheckpointOptions
    
    Adds `CheckpointOptions` to the triggered checkpoint messages (coordinator
    to barrier injecting tasks) and barriers (flowing inline with the data:
    
    ```java
    public class CheckpointOptions {
    
      // Type of checkpoint
      // => FULL_CHECKPOINT
      // => SAVEPOINT
      @NonNull
      CheckpointType getCheckpointType();
    
      // Custom target location. This is a String, because for future
      // backends it can be a logical location like a DB table.
      @Nullable
      String getTargetLocation();
    
    }
    ```
    
    This class would be the place to define more options for performing the
    checkpoints (for example for incremental checkpoints).
    
    These options are forwarded via the `StreamTask` to the `StreamOperator`s and
    `Snapshotable` backends. The `AbstractStreamOperator` checks the options and
    either i) forwards the shared per operator `CheckpointStreamFactory` (as of
    
    For this, the state backends provide the following new method:
    
    ```
    CheckpointStreamFactory createSavepointStreamFactory(JobID, String, String);
    ```
    
    The `MemoryStateBackend` returns the regular stream factory and the
    `FsStateBackend` returns a `FsSavepointStreamFactory`, which writes all
    checkpoint streams to a single directory (instead of the regular sub folders
    per checkpoint).
    
    We end up with the following directory layout for savepoints:
    
    ```
    +---------------------------+
    | :root_savepoint_directory | (custom per savepoint or configured default via `state.savepoints.dir`)
    +---------------------------+
      | +---------------------------------------+
      +-| savepoint-:jobId(0, 6)-:random_suffix | (one directory per savepoint)
        +---------------------------------------+
           |
           +- _metadata (one per savepoint)
           +- :uuid (one data file per StreamTask)
           +- ...
           +- :uuid
    ```

commit ca80d68ab7faba663f4c384156a90d58de0ebf82
Author: Ufuk Celebi <uc...@apache.org>
Date:   2017-02-16T16:56:37Z

    [FLINK-5763] [checkpoints] Adjust tests

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3346: [FLINK-5763] [checkpoints] Add CheckpointOptions for self...

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:

    https://github.com/apache/flink/pull/3346
  
    @uce I think you can close this PR. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3346: Selfcontained

Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3346#discussion_r101761879
  
    --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/AbstractStreamOperator.java ---
    @@ -431,11 +436,12 @@ public void snapshotState(StateSnapshotContext context) throws Exception {
     	@SuppressWarnings("deprecation")
     	@Deprecated
     	@Override
    -	public StreamStateHandle snapshotLegacyOperatorState(long checkpointId, long timestamp) throws Exception {
    +	public StreamStateHandle snapshotLegacyOperatorState(long checkpointId, long timestamp, CheckpointOptions checkpointOptions) throws Exception {
     		if (this instanceof StreamCheckpointedOperator) {
    +			CheckpointStreamFactory factory = getCheckpointStreamFactory(checkpointOptions);
    --- End diff --
    
    Here we create the factory again for legacy state. As an alternative, we would have to create the factory in the `StreamTask` and pass it to the snapshot call here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3346: [FLINK-5763] [checkpoints] Add CheckpointOptions for self...

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:

    https://github.com/apache/flink/pull/3346
  
    This is a prerequisite for https://issues.apache.org/jira/browse/FLINK-5820
    Reviewing this now...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3346: [FLINK-5763] [checkpoints] Add CheckpointOptions f...

Posted by uce <gi...@git.apache.org>.
Github user uce closed the pull request at:

    https://github.com/apache/flink/pull/3346


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3346: [FLINK-5763] [checkpoints] Add CheckpointOptions f...

Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3346#discussion_r102472651
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java ---
    @@ -296,15 +298,42 @@ public boolean isShutdown() {
     		checkNotNull(targetDirectory, "Savepoint target directory");
     
     		CheckpointProperties props = CheckpointProperties.forStandardSavepoint();
    -		CheckpointTriggerResult result = triggerCheckpoint(timestamp, props, targetDirectory, false);
     
    -		if (result.isSuccess()) {
    -			return result.getPendingCheckpoint().getCompletionFuture();
    -		}
    -		else {
    -			Throwable cause = new Exception("Failed to trigger savepoint: " + result.getFailureReason().message());
    -			return FlinkCompletableFuture.completedExceptionally(cause);
    +		// Create the unique savepoint directory
    +		final String savepointDirectory = SavepointStore
    --- End diff --
    
    Let's keep it as it is for this PR, will probably change a bit once the for the JobManager knows the StateBackend,


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3346: [FLINK-5763] [checkpoints] Add CheckpointOptions for self...

Posted by gyfora <gi...@git.apache.org>.
Github user gyfora commented on the issue:

    https://github.com/apache/flink/pull/3346
  
    So will the same savepoint logic apply to externalized checkpoints? I think it would be good to have a similar way of restoring from checkpoints and savepoints from a usability perspective. 
    
    I usually set the externalized checkpoint dir the same as the savepoint dir, to make it easy to write scripts to get the latest and restart.
    
    Otherwise I think the changes make a lot of sense and the directory structure looks very reasonable. 
    
    One question, in the directory name could we maybe use the checkpoint id instead of the random suffix or something more predictable? Maybe the checkpoint date?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3346: Selfcontained

Posted by uce <gi...@git.apache.org>.
Github user uce commented on a diff in the pull request:

    https://github.com/apache/flink/pull/3346#discussion_r101760988
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java ---
    @@ -296,15 +298,42 @@ public boolean isShutdown() {
     		checkNotNull(targetDirectory, "Savepoint target directory");
     
     		CheckpointProperties props = CheckpointProperties.forStandardSavepoint();
    -		CheckpointTriggerResult result = triggerCheckpoint(timestamp, props, targetDirectory, false);
     
    -		if (result.isSuccess()) {
    -			return result.getPendingCheckpoint().getCompletionFuture();
    -		}
    -		else {
    -			Throwable cause = new Exception("Failed to trigger savepoint: " + result.getFailureReason().message());
    -			return FlinkCompletableFuture.completedExceptionally(cause);
    +		// Create the unique savepoint directory
    +		final String savepointDirectory = SavepointStore
    --- End diff --
    
    We could also not create the directory here, but instead simply create the path name and send it to the tasks.
    
    @StephanEwen Maybe creating directories like these is something we should keep in mind for the state backends on the JobManager?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---