You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/06/08 10:25:46 UTC

[GitHub] [flink] fredia opened a new pull request, #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

fredia opened a new pull request, #19907:
URL: https://github.com/apache/flink/pull/19907

   <!--
   *Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
   
   *Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
   
   ## Contribution Checklist
   
     - Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
     
     - Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
     Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
     
     - Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
   
     - Each pull request should address only one issue, not mix up code from multiple issues.
     
     - Each commit in the pull request has a meaningful commit message (including the JIRA id)
   
     - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
   
   
   **(The sections below can be removed for hotfixes of typos)**
   -->
   
   ## What is the purpose of the change
   
   *Support local recovery for the materialized part of changelog.*
   
   ## Brief change log
   
     - *When doing snapshot, let `ChangelogKeyedStateBackend` return local materialized part from delegate backend.*
     - *Modify `TaskLocalStateStoreImpl` to maintain the relationship between local checkpoints.*
     - *Modify the logic of `confirmCheckpoint()` in `TaskLocalStateStoreImpl` to prevent the materialized part from being discarded.*
     -  *Modify `MiniCluster` to support working-dir.*
    
   
   ## Verifying this change
   
   Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing
   
   This change added tests and can be verified as follows:
   
     - *Added `ChangelogLocalRecoveryITCase`.*
     - *Extended `TaskLocalStateStoreImplTest` for confirming checkpoint and aborting checkpoint.*
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (no)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
     - The serializers: (no)
     - The runtime per-record code paths (performance sensitive): (no)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes, make changelog support local recovery.)
     - The S3 file system connector: (no)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (no)
     - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r928365136


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendLocalHandle.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.changelog;
+
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.SharedStateRegistry;
+import org.apache.flink.runtime.state.StateHandleID;
+
+import javax.annotation.Nullable;
+
+import java.util.List;
+
+/**
+ * State handle for local copies of {@link ChangelogStateHandleStreamImpl}. Consists of a
+ * remoteHandle that maintains the mapping of local handle and remote handle, like
+ * sharedStateHandleIDs in {@link org.apache.flink.runtime.state.IncrementalLocalKeyedStateHandle}.
+ */
+public class ChangelogStateBackendLocalHandle implements ChangelogStateBackendHandle {
+    private static final long serialVersionUID = 1L;
+    private final List<KeyedStateHandle> localMaterialized;
+    private final List<ChangelogStateHandle> localNonMaterialized;
+    private final ChangelogStateBackendHandleImpl remoteHandle;
+
+    public ChangelogStateBackendLocalHandle(
+            List<KeyedStateHandle> localMaterialized,
+            List<ChangelogStateHandle> localNonMaterialized,
+            ChangelogStateBackendHandleImpl remoteHandle) {
+        this.localMaterialized = localMaterialized;
+        this.localNonMaterialized = localNonMaterialized;
+        this.remoteHandle = remoteHandle;
+    }
+
+    @Override
+    public List<KeyedStateHandle> getMaterializedStateHandles() {
+        return localMaterialized;
+    }
+
+    @Override
+    public List<ChangelogStateHandle> getNonMaterializedStateHandles() {
+        return localNonMaterialized;
+    }
+
+    @Override
+    public long getMaterializationID() {
+        return remoteHandle.getMaterializationID();
+    }
+
+    @Override
+    public ChangelogStateBackendHandle rebound(long checkpointId) {
+        throw new UnsupportedOperationException("Should not call here.");
+    }
+
+    public List<KeyedStateHandle> getRemoteMaterializedStateHandles() {
+        return remoteHandle.getMaterializedStateHandles();
+    }
+
+    public List<ChangelogStateHandle> getRemoteNonMaterializedStateHandles() {
+        return remoteHandle.getNonMaterializedStateHandles();
+    }
+
+    @Override
+    public long getCheckpointId() {
+        return remoteHandle.getCheckpointId();
+    }
+
+    @Override
+    public void registerSharedStates(SharedStateRegistry stateRegistry, long checkpointID) {
+        remoteHandle.registerSharedStates(stateRegistry, checkpointID);
+    }
+
+    @Override
+    public long getCheckpointedSize() {
+        return remoteHandle.getCheckpointedSize();
+    }
+
+    @Override
+    public KeyGroupRange getKeyGroupRange() {
+        return remoteHandle.getKeyGroupRange();
+    }
+
+    @Nullable
+    @Override
+    public KeyedStateHandle getIntersection(KeyGroupRange keyGroupRange) {
+        throw new UnsupportedOperationException(
+                "This is a local state handle for the TM side only.");
+    }
+
+    @Override
+    public StateHandleID getStateHandleId() {
+        return remoteHandle.getStateHandleId();
+    }
+
+    @Override
+    public void discardState() throws Exception {
+        for (KeyedStateHandle keyedStateHandle : localMaterialized) {
+            keyedStateHandle.discardState();
+        }

Review Comment:
   Yes, It would be used by subsequent checkpoints and can't be deleted here.
   Fix `discardState()` as no-op.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] XComp commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
XComp commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r903346279


##########
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java:
##########
@@ -685,13 +687,27 @@ public CompletableFuture<Acknowledge> submitTask(
                     jobManagerConnection.getClassLoaderHandle();
             PartitionProducerStateChecker partitionStateChecker =
                     jobManagerConnection.getPartitionStateChecker();
+            TernaryBoolean changelogEnabled = TernaryBoolean.FALSE;
+            try {
+                changelogEnabled =
+                        InstantiationUtil.readObjectFromConfig(
+                                taskInformation.getTaskConfiguration(),
+                                "enablechangelog",
+                                getClass().getClassLoader());

Review Comment:
   That sounds reasonable in my opinion. Already better than having a hard-coded String representation here. 
   I'm just wondering whether config-related stuff should be better of in `flink-core` which is also a dependency of `flink-stream-java`.
   
   I'd imagine that @zentol might have an opinion here as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] pnowojski commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
pnowojski commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r903436217


##########
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java:
##########
@@ -685,13 +687,27 @@ public CompletableFuture<Acknowledge> submitTask(
                     jobManagerConnection.getClassLoaderHandle();
             PartitionProducerStateChecker partitionStateChecker =
                     jobManagerConnection.getPartitionStateChecker();
+            TernaryBoolean changelogEnabled = TernaryBoolean.FALSE;
+            try {
+                changelogEnabled =
+                        InstantiationUtil.readObjectFromConfig(
+                                taskInformation.getTaskConfiguration(),
+                                "enablechangelog",
+                                getClass().getClassLoader());

Review Comment:
   I'm not sure, but I think the idea sometime ago was to make "streaming api" runtime free. There were also independent plans of merging streaming runtime and batch runtime (or basically getting rid of batch runtime with dropping `DataSet API`). All in all it seems to me like `flink-core` would be a better place for config options? Indeed it would be nice to hear what @zentol thinks about it, as I presume/hope he has a better knowledge about those issues.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r928369565


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -186,20 +187,27 @@ private void persistLocalStateMetadata(long checkpointId, TaskStateSnapshot loca
 
     @VisibleForTesting
     File getTaskStateSnapshotFile(long checkpointId) {
-        final File checkpointDirectory =
-                localRecoveryConfig
-                        .getLocalStateDirectoryProvider()
-                        .orElseThrow(
-                                () -> new IllegalStateException("Local recovery must be enabled."))
-                        .subtaskSpecificCheckpointDirectory(checkpointId);
+        return new File(getCheckpointDirectory(checkpointId), TASK_STATE_SNAPSHOT_FILENAME);
+    }
 
+    private File createTaskStateSnapshotFile(long checkpointId) {

Review Comment:
   Nice suggestion. I have changed it into(the parameters are a little bit different ):
   - `getTaskStateSnapshotFile(long checkpointId)` // add filename suffix
   - `getCheckpointDirectory(checkpointId)` // as is
   - `createFolderOrFail(File checkpointDirectory)` // mkdirs



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] flinkbot commented on pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #19907:
URL: https://github.com/apache/flink/pull/19907#issuecomment-1149744705

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "038983ce1ae5dece534cc43db701450ca95b571c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "038983ce1ae5dece534cc43db701450ca95b571c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 038983ce1ae5dece534cc43db701450ca95b571c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926288182


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))
+                        .subtaskBaseDirectory(checkpointId);
+        File directoryForChangelog =
+                new File(checkpointDirectory, CHANGE_LOG_CHECKPOINT_PREFIX + checkpointId);
+
+        if (!directoryForChangelog.exists() && !directoryForChangelog.mkdirs()) {
+            throw new FlinkRuntimeException(
+                    String.format(
+                            "Could not create the checkpoint directory '%s'",
+                            directoryForChangelog));
+        }
+
+        return directoryForChangelog;
+    }
+
+    private void deleteMaterialization(LongPredicate pruningChecker) {
+        final Set<Long> materializationToRemove = new HashSet<>();
+        synchronized (lock) {
+            Iterator<Entry<Long, Long>> iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                long entryCheckpointId = entry.getKey();
+                if (pruningChecker.test(entryCheckpointId)) {
+                    materializationToRemove.add(entry.getValue());
+                    iterator.remove();
+                }
+            }
+
+            iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                materializationToRemove.remove(entry.getValue());
+            }
+        }
+
+        for (Long materializationId : materializationToRemove) {
+            File materializedDir =
+                    localRecoveryConfig
+                            .getLocalStateDirectoryProvider()
+                            .orElseThrow(
+                                    () ->
+                                            new IllegalStateException(
+                                                    "Local recovery must be enabled."))
+                            .subtaskSpecificCheckpointDirectory(materializationId);
+
+            if (!materializedDir.exists()) {
+                continue;
+            }
+            try {
+                deleteDirectory(materializedDir);
+            } catch (IOException ex) {
+                LOG.warn(
+                        "Exception while deleting local state directory of materialized part {} in subtask ({} - {} - {}).",
+                        materializedDir,
+                        jobID,
+                        jobVertexID,
+                        subtaskIndex,
+                        ex);
+            }
+        }
+    }
+
+    @Override
+    public void confirmCheckpoint(long confirmedCheckpointId) {
+        // Scenarios:
+        //   c1,m1
+        //   confirm c1, do nothing.
+        //   c2,m1
+        //   confirm c2, delete c1, don't delete m1
+        //   c3,m2
+        //   confirm c3, delete c2, delete m1
+        LOG.debug(
+                "Received confirmation for checkpoint {} in subtask ({} - {} - {}). Starting to prune history.",
+                confirmedCheckpointId,
+                jobID,
+                jobVertexID,
+                subtaskIndex);
+        // delete changelog-chk
+        pruneCheckpoints(checkpointId -> checkpointId < confirmedCheckpointId, false);

Review Comment:
   Fixed.  It was a mistake, but not break correctness. I think `breakOnceCheckerFalse` is used to reduce the number of iterations. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r920723644


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendLocalHandle.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.changelog;
+
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.SharedStateRegistry;
+import org.apache.flink.runtime.state.StateHandleID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.List;
+
+public class ChangelogStateBackendLocalHandle implements ChangelogStateBackendHandle {
+    private static final long serialVersionUID = 1L;
+    private static final Logger LOG =

Review Comment:
   It's used by `ChangelogKeyedStateBackend#completeRestore` to maintain a mapping from local to remote, like `sharedStateHandleIDs` in `IncrementalLocalKeyedStateHandle`.  The state handle in one checkpoint must be remote, if we restore from a local handle and do an incremental snapshot immediately, the base handles of the snapshot should refer to remote instead of local.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r928365724


##########
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java:
##########
@@ -138,12 +140,8 @@
 import org.apache.flink.util.concurrent.ExecutorThreadFactory;
 import org.apache.flink.util.concurrent.FutureUtils;
 
-import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
-import org.apache.flink.shaded.guava30.com.google.common.collect.Sets;
-
 import javax.annotation.Nonnull;
 import javax.annotation.Nullable;
-
 import java.io.File;

Review Comment:
   Got it, thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r920745521


##########
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java:
##########
@@ -685,13 +687,27 @@ public CompletableFuture<Acknowledge> submitTask(
                     jobManagerConnection.getClassLoaderHandle();
             PartitionProducerStateChecker partitionStateChecker =
                     jobManagerConnection.getPartitionStateChecker();
+            TernaryBoolean changelogEnabled = TernaryBoolean.FALSE;
+            try {
+                changelogEnabled =
+                        InstantiationUtil.readObjectFromConfig(
+                                taskInformation.getTaskConfiguration(),
+                                "enablechangelog",
+                                getClass().getClassLoader());

Review Comment:
   @rkhachatryan  I rebased https://github.com/apache/flink/pull/20160 and modified this part, could you please take a review?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926684030


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))

Review Comment:
   Also on line 340:
   ```
   LocalRecoveryDirectoryProvider directoryProvider =
           localRecoveryConfig
                   .getLocalStateDirectoryProvider()
                   .orElseThrow(LocalRecoveryConfig.localRecoveryNotEnabled());
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r895604615


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStore.java:
##########
@@ -76,7 +74,7 @@ public interface TaskLocalStateStore {
     /**
      * Remove all checkpoints from the store that match the given predicate.
      *
-     * @param matcher the predicate that selects the checkpoints for pruning.
+     * @param checkpointID the checkpoints for pruning.
      */
-    void pruneMatchingCheckpoints(LongPredicate matcher);
+    void pruneNotMatchingCheckpoints(long checkpointID);

Review Comment:
   It is not a pure refactoring.
   This method is called by `TaskStateManagerImpl#prioritizedOperatorState()` to prune other checkpoint when restoring. For changelog, the materialized part shouldn't be pruned when restoring, the `@param checkpointID` is used to find the reference between checkpoints.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on PR #19907:
URL: https://github.com/apache/flink/pull/19907#issuecomment-1169915049

   Thanks everyone for the discussion and review, I have created another ticket [FLINK-28286](https://issues.apache.org/jira/browse/FLINK-28286) for `ENABLE_CHANGE_LOG_STATE_BACKEND`, and addressed other comments, please let me know if you have any further concern.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r920731963


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendLocalHandle.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.changelog;
+
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.SharedStateRegistry;
+import org.apache.flink.runtime.state.StateHandleID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.List;
+
+public class ChangelogStateBackendLocalHandle implements ChangelogStateBackendHandle {
+    private static final long serialVersionUID = 1L;
+    private static final Logger LOG =
+            LoggerFactory.getLogger(ChangelogStateBackendLocalHandle.class);
+    private final List<KeyedStateHandle> localMaterialized;
+    private final List<ChangelogStateHandle> localNonMaterialized;
+    private final ChangelogStateBackendHandleImpl remoteHandle;
+
+    public ChangelogStateBackendLocalHandle(
+            List<KeyedStateHandle> localMaterialized,
+            List<ChangelogStateHandle> localNonMaterialized,
+            ChangelogStateBackendHandleImpl remoteHandle) {
+        this.localMaterialized = localMaterialized;
+        this.localNonMaterialized = localNonMaterialized;
+        this.remoteHandle = remoteHandle;
+    }
+
+    @Override
+    public List<KeyedStateHandle> getMaterializedStateHandles() {
+        return localMaterialized;
+    }
+
+    @Override
+    public List<ChangelogStateHandle> getNonMaterializedStateHandles() {
+        return localNonMaterialized;
+    }
+
+    @Override
+    public long getMaterializationID() {
+        return remoteHandle.getMaterializationID();
+    }
+
+    @Override
+    public ChangelogStateBackendHandle rebound(long checkpointId) {

Review Comment:
   >  changing config will not triggers local recovery and the rebound method will not be called.
   
   How about throw `UnsupportedOperationException("Should not call here.")` here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] masteryhx commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
masteryhx commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r920736651


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendLocalHandle.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.changelog;
+
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.SharedStateRegistry;
+import org.apache.flink.runtime.state.StateHandleID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.List;
+
+public class ChangelogStateBackendLocalHandle implements ChangelogStateBackendHandle {
+    private static final long serialVersionUID = 1L;
+    private static final Logger LOG =
+            LoggerFactory.getLogger(ChangelogStateBackendLocalHandle.class);
+    private final List<KeyedStateHandle> localMaterialized;
+    private final List<ChangelogStateHandle> localNonMaterialized;
+    private final ChangelogStateBackendHandleImpl remoteHandle;
+
+    public ChangelogStateBackendLocalHandle(
+            List<KeyedStateHandle> localMaterialized,
+            List<ChangelogStateHandle> localNonMaterialized,
+            ChangelogStateBackendHandleImpl remoteHandle) {
+        this.localMaterialized = localMaterialized;
+        this.localNonMaterialized = localNonMaterialized;
+        this.remoteHandle = remoteHandle;
+    }
+
+    @Override
+    public List<KeyedStateHandle> getMaterializedStateHandles() {
+        return localMaterialized;
+    }
+
+    @Override
+    public List<ChangelogStateHandle> getNonMaterializedStateHandles() {
+        return localNonMaterialized;
+    }
+
+    @Override
+    public long getMaterializationID() {
+        return remoteHandle.getMaterializationID();
+    }
+
+    @Override
+    public ChangelogStateBackendHandle rebound(long checkpointId) {

Review Comment:
   It makes sense to me currently.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r920723644


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendLocalHandle.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.changelog;
+
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.SharedStateRegistry;
+import org.apache.flink.runtime.state.StateHandleID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.List;
+
+public class ChangelogStateBackendLocalHandle implements ChangelogStateBackendHandle {
+    private static final long serialVersionUID = 1L;
+    private static final Logger LOG =

Review Comment:
   It's used by `ChangelogKeyedStateBackend#completeRestore` to maintain a mapping from local to remote, like `sharedStateHandleIDs` in `IncrementalLocalKeyedStateHandle`.  The state handle in one checkpoint must be remote, if we restore from a local handle and do an incremental snapshot immediately, the base handles of the snapshot should refer to remote instead of local.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] masteryhx commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
masteryhx commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r916507229


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendLocalHandle.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.changelog;
+
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.SharedStateRegistry;
+import org.apache.flink.runtime.state.StateHandleID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.List;
+
+public class ChangelogStateBackendLocalHandle implements ChangelogStateBackendHandle {
+    private static final long serialVersionUID = 1L;
+    private static final Logger LOG =
+            LoggerFactory.getLogger(ChangelogStateBackendLocalHandle.class);
+    private final List<KeyedStateHandle> localMaterialized;
+    private final List<ChangelogStateHandle> localNonMaterialized;
+    private final ChangelogStateBackendHandleImpl remoteHandle;
+
+    public ChangelogStateBackendLocalHandle(
+            List<KeyedStateHandle> localMaterialized,
+            List<ChangelogStateHandle> localNonMaterialized,
+            ChangelogStateBackendHandleImpl remoteHandle) {
+        this.localMaterialized = localMaterialized;
+        this.localNonMaterialized = localNonMaterialized;
+        this.remoteHandle = remoteHandle;
+    }
+
+    @Override
+    public List<KeyedStateHandle> getMaterializedStateHandles() {
+        return localMaterialized;
+    }
+
+    @Override
+    public List<ChangelogStateHandle> getNonMaterializedStateHandles() {
+        return localNonMaterialized;
+    }
+
+    @Override
+    public long getMaterializationID() {
+        return remoteHandle.getMaterializationID();
+    }
+
+    @Override
+    public ChangelogStateBackendHandle rebound(long checkpointId) {

Review Comment:
   In the current implementation, it will always use remote handle to recovery while changing enabled to disabled.
   See `DeactivatedChangelogStateBackend#restore`.
   Maybe we could rebound localMaterialized and localNonMaterialized and return ChangelogStateBackendLocalHandle. 
   Although currently changing config will not triggers local recovery and the rebound method will not be called, it will make the program more robust. WDYT ?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendLocalHandle.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.changelog;
+
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.SharedStateRegistry;
+import org.apache.flink.runtime.state.StateHandleID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.List;
+
+public class ChangelogStateBackendLocalHandle implements ChangelogStateBackendHandle {

Review Comment:
   nit: some comments for the class



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendLocalHandle.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.changelog;
+
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.SharedStateRegistry;
+import org.apache.flink.runtime.state.StateHandleID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.List;
+
+public class ChangelogStateBackendLocalHandle implements ChangelogStateBackendHandle {
+    private static final long serialVersionUID = 1L;
+    private static final Logger LOG =

Review Comment:
   It's not used in current class ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r895925820


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -141,6 +158,29 @@ public void storeLocalState(
                     subtaskIndex);
         }
 
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (currentMaterializationID == null
+                            || materializationID != currentMaterializationID.f0) {
+                        currentMaterializationID = Tuple2.of(materializationID, checkpointId);
+                        referredByCheckpointID.clear();

Review Comment:
   I add `lastCheckpointId` to check whether checkpoint is out of order.
   - if yes, don't write to local store, return directly. According to the materialization of the out-of-order checkpoint, there are two cases:
        - if cp with a newer materialization, say, (1)cp1 with m1, (2) cp3 with m2, (3) cp2 with m1.  when confirming cp3, cp1 and m1 would be discarded, and cp2 would not write to local store. therefore, no additional checkpoints are kept apart from cp3 and m2.
        -  if cp without a newer materialization, say, (1)cp1 with m1, (2) cp3 with m1, (3) cp2 with m1. when confirming cp3, m1 would retain, and cp2 would not write to local store. finally, m1,cp3 are kept.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926233838


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))

Review Comment:
   Extracted this into `getLocalRecoveryDirectoryProvider()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926149940


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))
+                        .subtaskBaseDirectory(checkpointId);
+        File directoryForChangelog =
+                new File(checkpointDirectory, CHANGE_LOG_CHECKPOINT_PREFIX + checkpointId);
+
+        if (!directoryForChangelog.exists() && !directoryForChangelog.mkdirs()) {
+            throw new FlinkRuntimeException(
+                    String.format(
+                            "Could not create the checkpoint directory '%s'",
+                            directoryForChangelog));
+        }
+
+        return directoryForChangelog;
+    }
+
+    private void deleteMaterialization(LongPredicate pruningChecker) {
+        final Set<Long> materializationToRemove = new HashSet<>();
+        synchronized (lock) {
+            Iterator<Entry<Long, Long>> iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                long entryCheckpointId = entry.getKey();
+                if (pruningChecker.test(entryCheckpointId)) {
+                    materializationToRemove.add(entry.getValue());
+                    iterator.remove();
+                }
+            }
+
+            iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                materializationToRemove.remove(entry.getValue());
+            }
+        }
+
+        for (Long materializationId : materializationToRemove) {
+            File materializedDir =
+                    localRecoveryConfig
+                            .getLocalStateDirectoryProvider()
+                            .orElseThrow(
+                                    () ->
+                                            new IllegalStateException(
+                                                    "Local recovery must be enabled."))
+                            .subtaskSpecificCheckpointDirectory(materializationId);
+
+            if (!materializedDir.exists()) {
+                continue;
+            }
+            try {
+                deleteDirectory(materializedDir);

Review Comment:
   Is it enough to delete the directory (even recursively)?
   Is it guaranteed that the wrapped backend uses only this folder for its local state?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))
+                        .subtaskBaseDirectory(checkpointId);
+        File directoryForChangelog =
+                new File(checkpointDirectory, CHANGE_LOG_CHECKPOINT_PREFIX + checkpointId);
+
+        if (!directoryForChangelog.exists() && !directoryForChangelog.mkdirs()) {
+            throw new FlinkRuntimeException(
+                    String.format(
+                            "Could not create the checkpoint directory '%s'",
+                            directoryForChangelog));
+        }
+
+        return directoryForChangelog;
+    }
+
+    private void deleteMaterialization(LongPredicate pruningChecker) {
+        final Set<Long> materializationToRemove = new HashSet<>();
+        synchronized (lock) {
+            Iterator<Entry<Long, Long>> iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                long entryCheckpointId = entry.getKey();
+                if (pruningChecker.test(entryCheckpointId)) {
+                    materializationToRemove.add(entry.getValue());
+                    iterator.remove();
+                }
+            }
+
+            iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                materializationToRemove.remove(entry.getValue());
+            }
+        }
+
+        for (Long materializationId : materializationToRemove) {
+            File materializedDir =
+                    localRecoveryConfig
+                            .getLocalStateDirectoryProvider()
+                            .orElseThrow(
+                                    () ->
+                                            new IllegalStateException(
+                                                    "Local recovery must be enabled."))
+                            .subtaskSpecificCheckpointDirectory(materializationId);

Review Comment:
   Can't this overlap with normal checkpoint folder?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926698315


##########
flink-tests/src/test/java/org/apache/flink/test/checkpointing/ChangelogLocalRecoveryITCase.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.checkpointing;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.changelog.fs.FsStateChangelogStorageFactory;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.StateChangelogOptions;
+import org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.minicluster.MiniCluster;
+import org.apache.flink.runtime.state.AbstractStateBackend;
+import org.apache.flink.runtime.state.StateBackend;
+import org.apache.flink.runtime.state.hashmap.HashMapStateBackend;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CollectionSink;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CountFunction;
+import org.apache.flink.test.util.InfiniteIntegerSource;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.File;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.configuration.CheckpointingOptions.LOCAL_RECOVERY;
+import static org.apache.flink.configuration.ClusterOptions.JOB_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.TASK_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning;
+
+/**
+ * Local recovery IT case for changelog. It never fails because local recovery is nice but not
+ * necessary.
+ */
+@RunWith(Parameterized.class)
+public class ChangelogLocalRecoveryITCase {
+
+    private static final int NUM_TASK_MANAGERS = 2;
+    private static final int NUM_TASK_SLOTS = 1;
+
+    @ClassRule public static final TemporaryFolder TEMPORARY_FOLDER = new TemporaryFolder();
+
+    @Parameterized.Parameter public AbstractStateBackend delegatedStateBackend;
+
+    @Parameterized.Parameters(name = "delegated state backend type = {0}")
+    public static Collection<AbstractStateBackend> parameter() {
+        return Arrays.asList(
+                new HashMapStateBackend(),
+                new EmbeddedRocksDBStateBackend(false),
+                new EmbeddedRocksDBStateBackend(true));
+    }
+
+    private MiniClusterWithClientResource cluster;
+    private static String workingDir;
+
+    @BeforeClass
+    public static void setWorkingDir() throws IOException {
+        workingDir = TEMPORARY_FOLDER.newFolder("work").getAbsolutePath();
+    }
+
+    @Before
+    public void setup() throws Exception {
+        Configuration configuration = new Configuration();
+        configuration.setInteger(CheckpointingOptions.MAX_RETAINED_CHECKPOINTS, 1);
+
+        configuration.setString(PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setString(JOB_MANAGER_PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setString(TASK_MANAGER_PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setBoolean(LOCAL_RECOVERY, true);
+        FsStateChangelogStorageFactory.configure(
+                configuration, TEMPORARY_FOLDER.newFolder(), Duration.ofMillis(1000), 1);
+        cluster =
+                new MiniClusterWithClientResource(
+                        new MiniClusterResourceConfiguration.Builder()
+                                .setConfiguration(configuration)
+                                .setNumberTaskManagers(NUM_TASK_MANAGERS)
+                                .setNumberSlotsPerTaskManager(NUM_TASK_SLOTS)
+                                .build());
+        cluster.before();
+        cluster.getMiniCluster().overrideRestoreModeForChangelogStateBackend();
+    }
+
+    private JobGraph buildJobGraph(StreamExecutionEnvironment env) {
+        env.addSource(new InfiniteIntegerSource())
+                .setParallelism(1)
+                .keyBy(element -> element)
+                .process(new CountFunction())
+                .addSink(new CollectionSink())
+                .setParallelism(1);
+        return env.getStreamGraph().getJobGraph();
+    }
+
+    @Test
+    public void testRestartTM() throws Exception {
+        File checkpointFolder = TEMPORARY_FOLDER.newFolder();
+        MiniCluster miniCluster = cluster.getMiniCluster();
+        StreamExecutionEnvironment env1 =
+                getEnv(delegatedStateBackend, checkpointFolder, true, 200, 800);
+        JobGraph firstJobGraph = buildJobGraph(env1);
+
+        miniCluster.submitJob(firstJobGraph).get();
+        waitForAllTaskRunning(miniCluster, firstJobGraph.getJobID(), false);
+        // wait job for doing materialization.
+        Thread.sleep(2000);

Review Comment:
   The updated version looks a bit fragile:
   ```String taskOwnedPath = checkpointPath + "/" + jobID.toString() + "/taskowned";```
   `taskowned` is an implementation detail and might change, breaking the test (or not breaking, while it should).
   
   Can't `ChangelogPeriodicMaterializationTestBase,getAllStateHandleId` be reused here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926713715


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))
+                        .subtaskBaseDirectory(checkpointId);
+        File directoryForChangelog =
+                new File(checkpointDirectory, CHANGE_LOG_CHECKPOINT_PREFIX + checkpointId);
+
+        if (!directoryForChangelog.exists() && !directoryForChangelog.mkdirs()) {
+            throw new FlinkRuntimeException(
+                    String.format(
+                            "Could not create the checkpoint directory '%s'",
+                            directoryForChangelog));
+        }
+
+        return directoryForChangelog;
+    }
+
+    private void deleteMaterialization(LongPredicate pruningChecker) {
+        final Set<Long> materializationToRemove = new HashSet<>();
+        synchronized (lock) {
+            Iterator<Entry<Long, Long>> iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                long entryCheckpointId = entry.getKey();
+                if (pruningChecker.test(entryCheckpointId)) {
+                    materializationToRemove.add(entry.getValue());
+                    iterator.remove();
+                }
+            }
+
+            iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                materializationToRemove.remove(entry.getValue());
+            }
+        }
+
+        for (Long materializationId : materializationToRemove) {
+            File materializedDir =
+                    localRecoveryConfig
+                            .getLocalStateDirectoryProvider()
+                            .orElseThrow(
+                                    () ->
+                                            new IllegalStateException(
+                                                    "Local recovery must be enabled."))
+                            .subtaskSpecificCheckpointDirectory(materializationId);
+
+            if (!materializedDir.exists()) {
+                continue;
+            }
+            try {
+                deleteDirectory(materializedDir);

Review Comment:
   The folder with the materialized state is discarded once no (changelog) checkpoint is using it.
   Conceptually, the handles could be discarded at that moment too, couldn't they?
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] masteryhx commented on pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
masteryhx commented on PR #19907:
URL: https://github.com/apache/flink/pull/19907#issuecomment-1153544582

   Thanks for the PR! I've left some comments, PTAL.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r928323324


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.containsKey(checkpointId)) {
+                        checkState(
+                                materializationID == mapToMaterializationId.get(checkpointId),
+                                "one checkpoint contains at most one materializationID");
+                    } else {
+                        mapToMaterializationId.put(checkpointId, materializationID);
+                    }
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        return new File(
+                getLocalRecoveryDirectoryProvider().subtaskBaseDirectory(checkpointId),
+                CHANGE_LOG_CHECKPOINT_PREFIX + checkpointId);
+    }
+
+    private void deleteMaterialization(LongPredicate pruningChecker) {
+        Set<Long> materializationToRemove;
+        synchronized (lock) {
+            Set<Long> checkpoints =
+                    mapToMaterializationId.keySet().stream()
+                            .filter(pruningChecker::test)
+                            .collect(Collectors.toSet());
+            materializationToRemove =
+                    checkpoints.stream()
+                            .map(mapToMaterializationId::remove)
+                            .collect(Collectors.toSet());
+            materializationToRemove.removeAll(mapToMaterializationId.values());
+        }
+
+        discardExecutor.execute(
+                () ->
+                        syncDiscardDirectoryForCollection(
+                                materializationToRemove.stream()
+                                        .map(
+                                                materializationId ->
+                                                        super.getCheckpointDirectory(
+                                                                materializationId))

Review Comment:
   nit: `.map(super::getCheckpointDirectory)`



##########
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java:
##########
@@ -138,12 +140,8 @@
 import org.apache.flink.util.concurrent.ExecutorThreadFactory;
 import org.apache.flink.util.concurrent.FutureUtils;
 
-import org.apache.flink.shaded.guava30.com.google.common.collect.ImmutableList;
-import org.apache.flink.shaded.guava30.com.google.common.collect.Sets;
-
 import javax.annotation.Nonnull;
 import javax.annotation.Nullable;
-
 import java.io.File;

Review Comment:
   Checkstyle failed because of imports of order.
   FYI: [build](https://dev.azure.com/khachatryanroman/flink/_build/results?buildId=1590&view=results) after fixing imports.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendLocalHandle.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.changelog;
+
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.SharedStateRegistry;
+import org.apache.flink.runtime.state.StateHandleID;
+
+import javax.annotation.Nullable;
+
+import java.util.List;
+
+/**
+ * State handle for local copies of {@link ChangelogStateHandleStreamImpl}. Consists of a
+ * remoteHandle that maintains the mapping of local handle and remote handle, like
+ * sharedStateHandleIDs in {@link org.apache.flink.runtime.state.IncrementalLocalKeyedStateHandle}.
+ */
+public class ChangelogStateBackendLocalHandle implements ChangelogStateBackendHandle {
+    private static final long serialVersionUID = 1L;
+    private final List<KeyedStateHandle> localMaterialized;
+    private final List<ChangelogStateHandle> localNonMaterialized;
+    private final ChangelogStateBackendHandleImpl remoteHandle;
+
+    public ChangelogStateBackendLocalHandle(
+            List<KeyedStateHandle> localMaterialized,
+            List<ChangelogStateHandle> localNonMaterialized,
+            ChangelogStateBackendHandleImpl remoteHandle) {
+        this.localMaterialized = localMaterialized;
+        this.localNonMaterialized = localNonMaterialized;
+        this.remoteHandle = remoteHandle;
+    }
+
+    @Override
+    public List<KeyedStateHandle> getMaterializedStateHandles() {
+        return localMaterialized;
+    }
+
+    @Override
+    public List<ChangelogStateHandle> getNonMaterializedStateHandles() {
+        return localNonMaterialized;
+    }
+
+    @Override
+    public long getMaterializationID() {
+        return remoteHandle.getMaterializationID();
+    }
+
+    @Override
+    public ChangelogStateBackendHandle rebound(long checkpointId) {
+        throw new UnsupportedOperationException("Should not call here.");
+    }
+
+    public List<KeyedStateHandle> getRemoteMaterializedStateHandles() {
+        return remoteHandle.getMaterializedStateHandles();
+    }
+
+    public List<ChangelogStateHandle> getRemoteNonMaterializedStateHandles() {
+        return remoteHandle.getNonMaterializedStateHandles();
+    }
+
+    @Override
+    public long getCheckpointId() {
+        return remoteHandle.getCheckpointId();
+    }
+
+    @Override
+    public void registerSharedStates(SharedStateRegistry stateRegistry, long checkpointID) {
+        remoteHandle.registerSharedStates(stateRegistry, checkpointID);
+    }
+
+    @Override
+    public long getCheckpointedSize() {
+        return remoteHandle.getCheckpointedSize();
+    }
+
+    @Override
+    public KeyGroupRange getKeyGroupRange() {
+        return remoteHandle.getKeyGroupRange();
+    }
+
+    @Nullable
+    @Override
+    public KeyedStateHandle getIntersection(KeyGroupRange keyGroupRange) {
+        throw new UnsupportedOperationException(
+                "This is a local state handle for the TM side only.");
+    }
+
+    @Override
+    public StateHandleID getStateHandleId() {
+        return remoteHandle.getStateHandleId();
+    }
+
+    @Override
+    public void discardState() throws Exception {
+        for (KeyedStateHandle keyedStateHandle : localMaterialized) {
+            keyedStateHandle.discardState();
+        }

Review Comment:
   Won't this discard materialized state potentially used by subsequent checkpoints?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -186,20 +187,27 @@ private void persistLocalStateMetadata(long checkpointId, TaskStateSnapshot loca
 
     @VisibleForTesting
     File getTaskStateSnapshotFile(long checkpointId) {
-        final File checkpointDirectory =
-                localRecoveryConfig
-                        .getLocalStateDirectoryProvider()
-                        .orElseThrow(
-                                () -> new IllegalStateException("Local recovery must be enabled."))
-                        .subtaskSpecificCheckpointDirectory(checkpointId);
+        return new File(getCheckpointDirectory(checkpointId), TASK_STATE_SNAPSHOT_FILENAME);
+    }
 
+    private File createTaskStateSnapshotFile(long checkpointId) {

Review Comment:
   The method doesn't actually create the file (rather the folder).
   
   I think the following set of methods would be more clear:
   - `getTaskStateSnapshotFile(String path, long checkpointId)` // add filename suffix
   - `getCheckpointDirectory(checkpointId)` // as is
   - `createFolderOrFail(String path)` // mkdirs
   
   WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan merged pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan merged PR #19907:
URL: https://github.com/apache/flink/pull/19907


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926118182


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -141,6 +158,29 @@ public void storeLocalState(
                     subtaskIndex);
         }
 
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (currentMaterializationID == null

Review Comment:
   1. Yes, but this method is called after the actual recovery - when reporting a new checkpoint
   2. Yes, but relying on it seems fragile
   
   How about adding an explicit check like this:
   ```
                       if (mapToMaterializationId.containsKey(checkpointId)) {
                           checkState(materializationID == mapToMaterializationId.get(checkpointId));
                       } else {
                           mapToMaterializationId.put(checkpointId, materializationID);
                       }
   ```
   
   and removing 
   ```
   if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
           < materializationID) {
       LOG.info(
   ```
   ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] zentol commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r908766389


##########
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java:
##########
@@ -685,13 +687,27 @@ public CompletableFuture<Acknowledge> submitTask(
                     jobManagerConnection.getClassLoaderHandle();
             PartitionProducerStateChecker partitionStateChecker =
                     jobManagerConnection.getPartitionStateChecker();
+            TernaryBoolean changelogEnabled = TernaryBoolean.FALSE;
+            try {
+                changelogEnabled =
+                        InstantiationUtil.readObjectFromConfig(
+                                taskInformation.getTaskConfiguration(),
+                                "enablechangelog",
+                                getClass().getClassLoader());

Review Comment:
   I would put it into `flink-core` personally; not sure if there is a huge benefit in spreading config options around.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r902531049


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/AbstractTaskLocalStateStore.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.util.AbstractMap;
+import java.util.AbstractMap.SimpleEntry;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Abstract TaskLocalStateStore. */
+public abstract class AbstractTaskLocalStateStore implements OwnedTaskLocalStateStore {

Review Comment:
   Because I thought they were two different implementations before. Yes, `AbstractTaskLocalStateStore` is not necessary, `ChangelogTaskLocalStateStore` can extend `TaskLocalStateStoreImpl` directly, I would revert it later.
    
   > And have you considered encapsulation instead of inheritance? Would it be more complex?
   
   Yes,  because `ChangelogTaskLocalStateStore` differs `TaskLocalStateStoreImpl` in `storeLocalState()`, `confirmCheckpoint()` and `abortCheckpoint()`, which needs to use some **private method** of `TaskLocalStateStoreImpl`, so I choosed inheritance instead of  encapsulation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926339002


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))
+                        .subtaskBaseDirectory(checkpointId);
+        File directoryForChangelog =
+                new File(checkpointDirectory, CHANGE_LOG_CHECKPOINT_PREFIX + checkpointId);
+
+        if (!directoryForChangelog.exists() && !directoryForChangelog.mkdirs()) {
+            throw new FlinkRuntimeException(
+                    String.format(
+                            "Could not create the checkpoint directory '%s'",
+                            directoryForChangelog));
+        }
+
+        return directoryForChangelog;
+    }
+
+    private void deleteMaterialization(LongPredicate pruningChecker) {
+        final Set<Long> materializationToRemove = new HashSet<>();
+        synchronized (lock) {
+            Iterator<Entry<Long, Long>> iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                long entryCheckpointId = entry.getKey();
+                if (pruningChecker.test(entryCheckpointId)) {
+                    materializationToRemove.add(entry.getValue());
+                    iterator.remove();
+                }
+            }
+
+            iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                materializationToRemove.remove(entry.getValue());
+            }
+        }
+
+        for (Long materializationId : materializationToRemove) {
+            File materializedDir =
+                    localRecoveryConfig
+                            .getLocalStateDirectoryProvider()
+                            .orElseThrow(
+                                    () ->
+                                            new IllegalStateException(
+                                                    "Local recovery must be enabled."))
+                            .subtaskSpecificCheckpointDirectory(materializationId);

Review Comment:
   Materialized part uses normal checkpoint folder(`chk_`), because the delegated state backend only knows normal checkpoint folder; snapshot meta(`_task_state_snapshot`) would use changelog folder(`changelog_chk_`). So, this would NOT overlap with normal checkpoint folder.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926698315


##########
flink-tests/src/test/java/org/apache/flink/test/checkpointing/ChangelogLocalRecoveryITCase.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.checkpointing;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.changelog.fs.FsStateChangelogStorageFactory;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.StateChangelogOptions;
+import org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.minicluster.MiniCluster;
+import org.apache.flink.runtime.state.AbstractStateBackend;
+import org.apache.flink.runtime.state.StateBackend;
+import org.apache.flink.runtime.state.hashmap.HashMapStateBackend;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CollectionSink;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CountFunction;
+import org.apache.flink.test.util.InfiniteIntegerSource;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.File;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.configuration.CheckpointingOptions.LOCAL_RECOVERY;
+import static org.apache.flink.configuration.ClusterOptions.JOB_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.TASK_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning;
+
+/**
+ * Local recovery IT case for changelog. It never fails because local recovery is nice but not
+ * necessary.
+ */
+@RunWith(Parameterized.class)
+public class ChangelogLocalRecoveryITCase {
+
+    private static final int NUM_TASK_MANAGERS = 2;
+    private static final int NUM_TASK_SLOTS = 1;
+
+    @ClassRule public static final TemporaryFolder TEMPORARY_FOLDER = new TemporaryFolder();
+
+    @Parameterized.Parameter public AbstractStateBackend delegatedStateBackend;
+
+    @Parameterized.Parameters(name = "delegated state backend type = {0}")
+    public static Collection<AbstractStateBackend> parameter() {
+        return Arrays.asList(
+                new HashMapStateBackend(),
+                new EmbeddedRocksDBStateBackend(false),
+                new EmbeddedRocksDBStateBackend(true));
+    }
+
+    private MiniClusterWithClientResource cluster;
+    private static String workingDir;
+
+    @BeforeClass
+    public static void setWorkingDir() throws IOException {
+        workingDir = TEMPORARY_FOLDER.newFolder("work").getAbsolutePath();
+    }
+
+    @Before
+    public void setup() throws Exception {
+        Configuration configuration = new Configuration();
+        configuration.setInteger(CheckpointingOptions.MAX_RETAINED_CHECKPOINTS, 1);
+
+        configuration.setString(PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setString(JOB_MANAGER_PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setString(TASK_MANAGER_PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setBoolean(LOCAL_RECOVERY, true);
+        FsStateChangelogStorageFactory.configure(
+                configuration, TEMPORARY_FOLDER.newFolder(), Duration.ofMillis(1000), 1);
+        cluster =
+                new MiniClusterWithClientResource(
+                        new MiniClusterResourceConfiguration.Builder()
+                                .setConfiguration(configuration)
+                                .setNumberTaskManagers(NUM_TASK_MANAGERS)
+                                .setNumberSlotsPerTaskManager(NUM_TASK_SLOTS)
+                                .build());
+        cluster.before();
+        cluster.getMiniCluster().overrideRestoreModeForChangelogStateBackend();
+    }
+
+    private JobGraph buildJobGraph(StreamExecutionEnvironment env) {
+        env.addSource(new InfiniteIntegerSource())
+                .setParallelism(1)
+                .keyBy(element -> element)
+                .process(new CountFunction())
+                .addSink(new CollectionSink())
+                .setParallelism(1);
+        return env.getStreamGraph().getJobGraph();
+    }
+
+    @Test
+    public void testRestartTM() throws Exception {
+        File checkpointFolder = TEMPORARY_FOLDER.newFolder();
+        MiniCluster miniCluster = cluster.getMiniCluster();
+        StreamExecutionEnvironment env1 =
+                getEnv(delegatedStateBackend, checkpointFolder, true, 200, 800);
+        JobGraph firstJobGraph = buildJobGraph(env1);
+
+        miniCluster.submitJob(firstJobGraph).get();
+        waitForAllTaskRunning(miniCluster, firstJobGraph.getJobID(), false);
+        // wait job for doing materialization.
+        Thread.sleep(2000);

Review Comment:
   The updated version looks a bit fragile:
   ```String taskOwnedPath = checkpointPath + "/" + jobID.toString() + "/taskowned";```
   
   Can't `ChangelogPeriodicMaterializationTestBase,getAllStateHandleId` be reused here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r902497128


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/AbstractTaskLocalStateStore.java:
##########
@@ -0,0 +1,430 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.util.AbstractMap;
+import java.util.AbstractMap.SimpleEntry;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Abstract TaskLocalStateStore. */
+public abstract class AbstractTaskLocalStateStore implements OwnedTaskLocalStateStore {

Review Comment:
   Could you explain why a separate `AbstractTaskLocalStateStore` class, not just `ChangelogTaskLocalStateStore` extending `TaskLocalStateStoreImpl`?
   
   And have you considered encapsulation instead of inheritance? Would it be more complex?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java:
##########
@@ -685,13 +687,27 @@ public CompletableFuture<Acknowledge> submitTask(
                     jobManagerConnection.getClassLoaderHandle();
             PartitionProducerStateChecker partitionStateChecker =
                     jobManagerConnection.getPartitionStateChecker();
+            TernaryBoolean changelogEnabled = TernaryBoolean.FALSE;
+            try {
+                changelogEnabled =
+                        InstantiationUtil.readObjectFromConfig(
+                                taskInformation.getTaskConfiguration(),
+                                "enablechangelog",
+                                getClass().getClassLoader());

Review Comment:
   I understand that the reason of not using `StreamConfig.ENABLE_CHANGE_LOG_STATE_BACKEND` constant is that it's in a different module (`flink-streaming-java`), not in `flink-runtime`.
   and `flink-runtime` should not depend on `flink-streaming-java`.
   
   However, the opposite is fine I think, `flink-streaming-java` can depend on `flink-runtime` (there were some plans to eventually merge them IIRC),
   so the constant `ENABLE_CHANGE_LOG_STATE_BACKEND` could be moved to `flink-runtime`.
   
   An obvious name would be `TaskConfig`, but it already exists (in `flink-runtime`) and is used for other purposes (other types of tasks).
   
   So how about adding `TaskExecutionConfig` class to `flink-runtime`, package `org.apache.flink.runtime.taskexecutor`, and placing the constant+parsing there?
   
   cc: @akalash, @XComp, @pnowojski
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926200002


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -409,39 +405,31 @@ private void discardLocalStateForCheckpoint(long checkpointID, Optional<TaskStat
                     }
                 });
 
-        Optional<LocalRecoveryDirectoryProvider> directoryProviderOptional =
-                localRecoveryConfig.getLocalStateDirectoryProvider();
+        File checkpointDir = getCheckpointDirectory(checkpointID);
 
-        if (directoryProviderOptional.isPresent()) {
-            File checkpointDir =
-                    directoryProviderOptional
-                            .get()
-                            .subtaskSpecificCheckpointDirectory(checkpointID);

Review Comment:
   > Is it intentional? If not, is it safe?
   
   Yes, this slightly changes the semantics, it is to reuse the logic of `getCheckpointDirectory`. It is safe, because the directory would be deleted by:
   ```
    try {
               deleteDirectory(checkpointDir);
           } catch (IOException ex) {
   ...
   }
   ```
   
   I will refactor the ` getCheckpointDirectory()` in https://issues.apache.org/jira/browse/FLINK-28614 , separate the logic of `get` and `create`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926305448


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -464,7 +452,7 @@ private void pruneCheckpoints(LongPredicate pruningChecker, boolean breakOnceChe
                 long entryCheckpointId = snapshotEntry.getKey();
 
                 if (pruningChecker.test(entryCheckpointId)) {
-                    toRemove.add(snapshotEntry);
+                    toRemove.add(new SimpleEntry<>(snapshotEntry));

Review Comment:
   It was a bug here, `snapshotEntry` would be updated during iterating, which would cause items already added to `toRemove` to be changed. 
   
   Nice suggestion, I have replaced `Map.Entry` with `Tuple2`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r895925820


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -141,6 +158,29 @@ public void storeLocalState(
                     subtaskIndex);
         }
 
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (currentMaterializationID == null
+                            || materializationID != currentMaterializationID.f0) {
+                        currentMaterializationID = Tuple2.of(materializationID, checkpointId);
+                        referredByCheckpointID.clear();

Review Comment:
   I add `lastCheckpointId` to check whether checkpoint is out of order.
   - if yes, don't write to local store, return directly. According to the materialization of the out-of-order checkpoint, there are two cases:
        - if cp with a newer materialization, say, (1)cp1 with m1, (2) cp3 with m2, (3) cp2 with m1.  when confirming cp3, cp1 and m1 would be discarded, and cp2 would not write to local store. therefore, no additional checkpoints are kept apart from cp3 and m2.
        -  if cp without a newer materialization, say, (1)cp1 with m1, (2) cp3 with m1, (3) cp2 with m1. when confirming cp3, cp1 would retain, and cp2 would not write to local store. finally, cp1,m1,cp3 are kept.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -98,6 +104,15 @@ public class TaskLocalStateStoreImpl implements OwnedTaskLocalStateStore {
     @GuardedBy("lock")
     private final SortedMap<Long, TaskStateSnapshot> storedTaskStateByCheckpointID;
 
+    /** The relationship between checkpoints. (cp1,cp3) means cp3 refer to cp1's state. */
+    private final Map<Long, Long> referredByCheckpointID;
+
+    /**
+     * (MaterializationID, created by checkpointID). Because local store only keeps one checkpoint,
+     * at most one materialization can exist at the same time.
+     */
+    private Tuple2<Long, Long> currentMaterializationID;

Review Comment:
   Nice suggestion, I created `ChangelogTaskLocalStateStore` to encapsulate changelog-specific logic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r895604615


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStore.java:
##########
@@ -76,7 +74,7 @@ public interface TaskLocalStateStore {
     /**
      * Remove all checkpoints from the store that match the given predicate.
      *
-     * @param matcher the predicate that selects the checkpoints for pruning.
+     * @param checkpointID the checkpoints for pruning.
      */
-    void pruneMatchingCheckpoints(LongPredicate matcher);
+    void pruneNotMatchingCheckpoints(long checkpointID);

Review Comment:
   Thanks for the suggestions, revert it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926672725


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))
+                        .subtaskBaseDirectory(checkpointId);
+        File directoryForChangelog =
+                new File(checkpointDirectory, CHANGE_LOG_CHECKPOINT_PREFIX + checkpointId);
+
+        if (!directoryForChangelog.exists() && !directoryForChangelog.mkdirs()) {
+            throw new FlinkRuntimeException(
+                    String.format(
+                            "Could not create the checkpoint directory '%s'",
+                            directoryForChangelog));
+        }
+
+        return directoryForChangelog;
+    }
+
+    private void deleteMaterialization(LongPredicate pruningChecker) {
+        final Set<Long> materializationToRemove = new HashSet<>();
+        synchronized (lock) {
+            Iterator<Entry<Long, Long>> iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                long entryCheckpointId = entry.getKey();
+                if (pruningChecker.test(entryCheckpointId)) {
+                    materializationToRemove.add(entry.getValue());
+                    iterator.remove();
+                }
+            }
+
+            iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                materializationToRemove.remove(entry.getValue());
+            }
+        }
+
+        for (Long materializationId : materializationToRemove) {
+            File materializedDir =
+                    localRecoveryConfig
+                            .getLocalStateDirectoryProvider()
+                            .orElseThrow(
+                                    () ->
+                                            new IllegalStateException(
+                                                    "Local recovery must be enabled."))
+                            .subtaskSpecificCheckpointDirectory(materializationId);
+
+            if (!materializedDir.exists()) {
+                continue;
+            }
+            try {
+                deleteDirectory(materializedDir);

Review Comment:
   Yes, but I mean other state backends.
   I'm concerned because the current deletion also involves explicit `discardState()` call on handles.
    



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926693644


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))
+                        .subtaskBaseDirectory(checkpointId);
+        File directoryForChangelog =
+                new File(checkpointDirectory, CHANGE_LOG_CHECKPOINT_PREFIX + checkpointId);
+
+        if (!directoryForChangelog.exists() && !directoryForChangelog.mkdirs()) {
+            throw new FlinkRuntimeException(
+                    String.format(
+                            "Could not create the checkpoint directory '%s'",
+                            directoryForChangelog));
+        }
+
+        return directoryForChangelog;
+    }
+
+    private void deleteMaterialization(LongPredicate pruningChecker) {
+        final Set<Long> materializationToRemove = new HashSet<>();
+        synchronized (lock) {
+            Iterator<Entry<Long, Long>> iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                long entryCheckpointId = entry.getKey();
+                if (pruningChecker.test(entryCheckpointId)) {
+                    materializationToRemove.add(entry.getValue());
+                    iterator.remove();
+                }
+            }
+
+            iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                materializationToRemove.remove(entry.getValue());
+            }
+        }
+
+        for (Long materializationId : materializationToRemove) {
+            File materializedDir =
+                    localRecoveryConfig
+                            .getLocalStateDirectoryProvider()
+                            .orElseThrow(
+                                    () ->
+                                            new IllegalStateException(
+                                                    "Local recovery must be enabled."))
+                            .subtaskSpecificCheckpointDirectory(materializationId);
+
+            if (!materializedDir.exists()) {
+                continue;
+            }
+            try {
+                deleteDirectory(materializedDir);

Review Comment:
   But we can‘t explicitly discard materialized state in `ChangelogStateBackendLocalHandle#discardState()`, because one materialized state is referenced by several changelog checkpoints. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926707252


##########
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java:
##########
@@ -686,12 +688,25 @@ public CompletableFuture<Acknowledge> submitTask(
             PartitionProducerStateChecker partitionStateChecker =
                     jobManagerConnection.getPartitionStateChecker();
 
+            // Configuration from application will override the one from env.
+            boolean envChangelogEnabled =
+                    taskManagerConfiguration
+                            .getConfiguration()

Review Comment:
   I meant something like this:
   ```
   // in TaskExecutor
   final TaskLocalStateStore localStateStore =
       localStateStoresManager.localStateStoreForSub
           jobId,
           tdd.getAllocationId(),
           taskInformation.getJobVertexId(),
           tdd.getSubtaskIndex(),
           taskManagerConfiguration.getConfigura
           jobInformation.getJobConfiguration())
   
   // TaskExecutorLocalStateStoresManager.localStateStoreForSubtask
       ...
       boolean changelogEnabled =
           jobConfiguration
               .getOptional(
                   StateChangelogOptionsInternal
                       .ENABLE_CHANGE_LOG_FOR_APPLICATION)
               .orElse(
                   clusterConfiguration.getBoolean(
                       StateChangelogOptions.ENABLE_STATE_CHANGE_LOG));
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926124747


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))

Review Comment:
   Should we extract this into a field or function already:) 
   
   (also duplicated in parent)



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))
+                        .subtaskBaseDirectory(checkpointId);
+        File directoryForChangelog =
+                new File(checkpointDirectory, CHANGE_LOG_CHECKPOINT_PREFIX + checkpointId);
+
+        if (!directoryForChangelog.exists() && !directoryForChangelog.mkdirs()) {
+            throw new FlinkRuntimeException(
+                    String.format(
+                            "Could not create the checkpoint directory '%s'",
+                            directoryForChangelog));
+        }
+
+        return directoryForChangelog;
+    }
+
+    private void deleteMaterialization(LongPredicate pruningChecker) {
+        final Set<Long> materializationToRemove = new HashSet<>();
+        synchronized (lock) {
+            Iterator<Entry<Long, Long>> iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                long entryCheckpointId = entry.getKey();
+                if (pruningChecker.test(entryCheckpointId)) {
+                    materializationToRemove.add(entry.getValue());
+                    iterator.remove();
+                }
+            }
+
+            iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                materializationToRemove.remove(entry.getValue());
+            }

Review Comment:
   How about simplifying this code?
   ```
   Set<Long> checkpoints =
           mapToMaterializationId.keySet().stream()
                   .filter(pruningChecker::test)
                   .collect(Collectors.toSet());
   materializationToRemove =
           checkpoints.stream()
                   .map(mapToMaterializationId::remove)
                   .collect(Collectors.toSet());
   materializationToRemove.removeAll(mapToMaterializationId.values());
   ```
   
   Alternatively, we could maintain a Map<materializationId, Set<checkpointId>> and discard materialization whenever the set is empty. But two maps seems more complex.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))
+                        .subtaskBaseDirectory(checkpointId);
+        File directoryForChangelog =
+                new File(checkpointDirectory, CHANGE_LOG_CHECKPOINT_PREFIX + checkpointId);
+
+        if (!directoryForChangelog.exists() && !directoryForChangelog.mkdirs()) {
+            throw new FlinkRuntimeException(
+                    String.format(
+                            "Could not create the checkpoint directory '%s'",
+                            directoryForChangelog));
+        }
+
+        return directoryForChangelog;
+    }
+
+    private void deleteMaterialization(LongPredicate pruningChecker) {
+        final Set<Long> materializationToRemove = new HashSet<>();
+        synchronized (lock) {
+            Iterator<Entry<Long, Long>> iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                long entryCheckpointId = entry.getKey();
+                if (pruningChecker.test(entryCheckpointId)) {
+                    materializationToRemove.add(entry.getValue());
+                    iterator.remove();
+                }
+            }
+
+            iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                materializationToRemove.remove(entry.getValue());
+            }
+        }
+
+        for (Long materializationId : materializationToRemove) {
+            File materializedDir =
+                    localRecoveryConfig
+                            .getLocalStateDirectoryProvider()
+                            .orElseThrow(
+                                    () ->
+                                            new IllegalStateException(
+                                                    "Local recovery must be enabled."))
+                            .subtaskSpecificCheckpointDirectory(materializationId);
+
+            if (!materializedDir.exists()) {
+                continue;
+            }
+            try {
+                deleteDirectory(materializedDir);
+            } catch (IOException ex) {
+                LOG.warn(
+                        "Exception while deleting local state directory of materialized part {} in subtask ({} - {} - {}).",
+                        materializedDir,
+                        jobID,
+                        jobVertexID,
+                        subtaskIndex,
+                        ex);
+            }
+        }
+    }
+
+    @Override
+    public void confirmCheckpoint(long confirmedCheckpointId) {

Review Comment:
   Instead of overriding (and duplicating) these two methdos (confirm & abort)
   can we override `pruneCheckpoints`?
   Deleting materialization seems to be a part of pruning...
   
   ```
       @Override
       protected void pruneCheckpoints(LongPredicate pruningChecker, boolean breakOnceCheckerFalse) {
           deleteMaterialization(pruningChecker);
           super.pruneCheckpoints(pruningChecker, breakOnceCheckerFalse);
       }
   ```



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))
+                        .subtaskBaseDirectory(checkpointId);
+        File directoryForChangelog =
+                new File(checkpointDirectory, CHANGE_LOG_CHECKPOINT_PREFIX + checkpointId);
+
+        if (!directoryForChangelog.exists() && !directoryForChangelog.mkdirs()) {
+            throw new FlinkRuntimeException(
+                    String.format(
+                            "Could not create the checkpoint directory '%s'",
+                            directoryForChangelog));
+        }
+
+        return directoryForChangelog;
+    }
+
+    private void deleteMaterialization(LongPredicate pruningChecker) {
+        final Set<Long> materializationToRemove = new HashSet<>();
+        synchronized (lock) {
+            Iterator<Entry<Long, Long>> iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                long entryCheckpointId = entry.getKey();
+                if (pruningChecker.test(entryCheckpointId)) {
+                    materializationToRemove.add(entry.getValue());
+                    iterator.remove();
+                }
+            }
+
+            iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                materializationToRemove.remove(entry.getValue());
+            }
+        }
+
+        for (Long materializationId : materializationToRemove) {
+            File materializedDir =
+                    localRecoveryConfig
+                            .getLocalStateDirectoryProvider()
+                            .orElseThrow(
+                                    () ->
+                                            new IllegalStateException(
+                                                    "Local recovery must be enabled."))
+                            .subtaskSpecificCheckpointDirectory(materializationId);
+
+            if (!materializedDir.exists()) {
+                continue;
+            }

Review Comment:
   nit: invert if?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskExecutorLocalStateStoresManager.java:
##########
@@ -164,22 +165,27 @@ public TaskLocalStateStore localStateStoreForSubtask(
                 LocalRecoveryConfig localRecoveryConfig =
                         new LocalRecoveryConfig(directoryProvider);
 
-                taskLocalStateStore =
-                        localRecoveryConfig.isLocalRecoveryEnabled()
-                                ?
-
-                                // Real store implementation if local recovery is enabled
-                                new TaskLocalStateStoreImpl(
-                                        jobId,
-                                        allocationID,
-                                        jobVertexID,
-                                        subtaskIndex,
-                                        localRecoveryConfig,
-                                        discardExecutor)
-                                :
-
-                                // NOP implementation if local recovery is disabled
-                                new NoOpTaskLocalStateStoreImpl(localRecoveryConfig);
+                if (localRecoveryConfig.isLocalRecoveryEnabled()) {
+                    taskLocalStateStore =
+                            changelogEnabled
+                                    ? new ChangelogTaskLocalStateStore(

Review Comment:
   nit: I'd flatten `if`s and have no-op branch followed by changelog branch followed by normal branch



##########
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java:
##########
@@ -686,12 +688,25 @@ public CompletableFuture<Acknowledge> submitTask(
             PartitionProducerStateChecker partitionStateChecker =
                     jobManagerConnection.getPartitionStateChecker();
 
+            // Configuration from application will override the one from env.
+            boolean envChangelogEnabled =
+                    taskManagerConfiguration
+                            .getConfiguration()

Review Comment:
   How about passing cluster and job configurations to `TaskExecutorLocalStateStoresManager.localStateStoreForSubtask`
   and parsing there?
   
   nit: I'd inline `envChangelogEnabled` but that's a matter of taste :)



##########
flink-tests/src/test/java/org/apache/flink/test/checkpointing/ChangelogLocalRecoveryITCase.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.checkpointing;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.changelog.fs.FsStateChangelogStorageFactory;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.StateChangelogOptions;
+import org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.minicluster.MiniCluster;
+import org.apache.flink.runtime.state.AbstractStateBackend;
+import org.apache.flink.runtime.state.StateBackend;
+import org.apache.flink.runtime.state.hashmap.HashMapStateBackend;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CollectionSink;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CountFunction;
+import org.apache.flink.test.util.InfiniteIntegerSource;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.File;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.configuration.CheckpointingOptions.LOCAL_RECOVERY;
+import static org.apache.flink.configuration.ClusterOptions.JOB_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.TASK_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning;
+
+/**
+ * Local recovery IT case for changelog. It never fails because local recovery is nice but not
+ * necessary.
+ */
+@RunWith(Parameterized.class)
+public class ChangelogLocalRecoveryITCase {

Review Comment:
   extend `TestLogger`?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -409,39 +405,31 @@ private void discardLocalStateForCheckpoint(long checkpointID, Optional<TaskStat
                     }
                 });
 
-        Optional<LocalRecoveryDirectoryProvider> directoryProviderOptional =
-                localRecoveryConfig.getLocalStateDirectoryProvider();
+        File checkpointDir = getCheckpointDirectory(checkpointID);
 
-        if (directoryProviderOptional.isPresent()) {
-            File checkpointDir =
-                    directoryProviderOptional
-                            .get()
-                            .subtaskSpecificCheckpointDirectory(checkpointID);

Review Comment:
   This slightly changes the semantics - it now tries to re-create the folder if it doesn't exist. Is it intentional? If not, is it safe?
   
   (asserting that the provider exists looks safe)



##########
flink-tests/src/test/java/org/apache/flink/test/checkpointing/ChangelogLocalRecoveryITCase.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.checkpointing;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.changelog.fs.FsStateChangelogStorageFactory;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.StateChangelogOptions;
+import org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.minicluster.MiniCluster;
+import org.apache.flink.runtime.state.AbstractStateBackend;
+import org.apache.flink.runtime.state.StateBackend;
+import org.apache.flink.runtime.state.hashmap.HashMapStateBackend;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CollectionSink;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CountFunction;
+import org.apache.flink.test.util.InfiniteIntegerSource;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.File;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.configuration.CheckpointingOptions.LOCAL_RECOVERY;
+import static org.apache.flink.configuration.ClusterOptions.JOB_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.TASK_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning;
+
+/**
+ * Local recovery IT case for changelog. It never fails because local recovery is nice but not
+ * necessary.
+ */
+@RunWith(Parameterized.class)
+public class ChangelogLocalRecoveryITCase {
+
+    private static final int NUM_TASK_MANAGERS = 2;
+    private static final int NUM_TASK_SLOTS = 1;
+
+    @ClassRule public static final TemporaryFolder TEMPORARY_FOLDER = new TemporaryFolder();
+
+    @Parameterized.Parameter public AbstractStateBackend delegatedStateBackend;
+
+    @Parameterized.Parameters(name = "delegated state backend type = {0}")
+    public static Collection<AbstractStateBackend> parameter() {
+        return Arrays.asList(
+                new HashMapStateBackend(),
+                new EmbeddedRocksDBStateBackend(false),
+                new EmbeddedRocksDBStateBackend(true));
+    }
+
+    private MiniClusterWithClientResource cluster;
+    private static String workingDir;
+
+    @BeforeClass
+    public static void setWorkingDir() throws IOException {
+        workingDir = TEMPORARY_FOLDER.newFolder("work").getAbsolutePath();
+    }
+
+    @Before
+    public void setup() throws Exception {
+        Configuration configuration = new Configuration();
+        configuration.setInteger(CheckpointingOptions.MAX_RETAINED_CHECKPOINTS, 1);
+
+        configuration.setString(PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setString(JOB_MANAGER_PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setString(TASK_MANAGER_PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setBoolean(LOCAL_RECOVERY, true);
+        FsStateChangelogStorageFactory.configure(
+                configuration, TEMPORARY_FOLDER.newFolder(), Duration.ofMillis(1000), 1);
+        cluster =
+                new MiniClusterWithClientResource(
+                        new MiniClusterResourceConfiguration.Builder()
+                                .setConfiguration(configuration)
+                                .setNumberTaskManagers(NUM_TASK_MANAGERS)
+                                .setNumberSlotsPerTaskManager(NUM_TASK_SLOTS)
+                                .build());
+        cluster.before();

Review Comment:
   I think cluster teardown is missing.
   
   nit: and probably we could use a rule for that (something like `@ClassRule ... MiniClusterWithClientResource miniClusterResource`).



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -464,7 +452,7 @@ private void pruneCheckpoints(LongPredicate pruningChecker, boolean breakOnceChe
                 long entryCheckpointId = snapshotEntry.getKey();
 
                 if (pruningChecker.test(entryCheckpointId)) {
-                    toRemove.add(snapshotEntry);
+                    toRemove.add(new SimpleEntry<>(snapshotEntry));

Review Comment:
   Please add a comment why it's necessary (or even better I guess refactor the code and replace `Map.Entry` with `Tuple2` but that's probably out of scope).



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))
+                        .subtaskBaseDirectory(checkpointId);
+        File directoryForChangelog =
+                new File(checkpointDirectory, CHANGE_LOG_CHECKPOINT_PREFIX + checkpointId);
+
+        if (!directoryForChangelog.exists() && !directoryForChangelog.mkdirs()) {
+            throw new FlinkRuntimeException(
+                    String.format(
+                            "Could not create the checkpoint directory '%s'",
+                            directoryForChangelog));
+        }
+
+        return directoryForChangelog;
+    }
+
+    private void deleteMaterialization(LongPredicate pruningChecker) {
+        final Set<Long> materializationToRemove = new HashSet<>();
+        synchronized (lock) {
+            Iterator<Entry<Long, Long>> iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                long entryCheckpointId = entry.getKey();
+                if (pruningChecker.test(entryCheckpointId)) {
+                    materializationToRemove.add(entry.getValue());
+                    iterator.remove();
+                }
+            }
+
+            iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                materializationToRemove.remove(entry.getValue());
+            }
+        }
+
+        for (Long materializationId : materializationToRemove) {
+            File materializedDir =
+                    localRecoveryConfig
+                            .getLocalStateDirectoryProvider()
+                            .orElseThrow(
+                                    () ->
+                                            new IllegalStateException(
+                                                    "Local recovery must be enabled."))
+                            .subtaskSpecificCheckpointDirectory(materializationId);
+
+            if (!materializedDir.exists()) {
+                continue;
+            }
+            try {
+                deleteDirectory(materializedDir);
+            } catch (IOException ex) {
+                LOG.warn(
+                        "Exception while deleting local state directory of materialized part {} in subtask ({} - {} - {}).",
+                        materializedDir,
+                        jobID,
+                        jobVertexID,
+                        subtaskIndex,
+                        ex);
+            }
+        }
+    }
+
+    @Override
+    public void confirmCheckpoint(long confirmedCheckpointId) {
+        // Scenarios:
+        //   c1,m1
+        //   confirm c1, do nothing.
+        //   c2,m1
+        //   confirm c2, delete c1, don't delete m1
+        //   c3,m2
+        //   confirm c3, delete c2, delete m1
+        LOG.debug(
+                "Received confirmation for checkpoint {} in subtask ({} - {} - {}). Starting to prune history.",
+                confirmedCheckpointId,
+                jobID,
+                jobVertexID,
+                subtaskIndex);
+        // delete changelog-chk
+        pruneCheckpoints(checkpointId -> checkpointId < confirmedCheckpointId, false);

Review Comment:
   Parent method has `true` here (break iteration).
   Is it intentional?



##########
flink-tests/src/test/java/org/apache/flink/test/checkpointing/ChangelogLocalRecoveryITCase.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.checkpointing;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.changelog.fs.FsStateChangelogStorageFactory;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.StateChangelogOptions;
+import org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.minicluster.MiniCluster;
+import org.apache.flink.runtime.state.AbstractStateBackend;
+import org.apache.flink.runtime.state.StateBackend;
+import org.apache.flink.runtime.state.hashmap.HashMapStateBackend;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CollectionSink;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CountFunction;
+import org.apache.flink.test.util.InfiniteIntegerSource;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.File;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.configuration.CheckpointingOptions.LOCAL_RECOVERY;
+import static org.apache.flink.configuration.ClusterOptions.JOB_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.TASK_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning;
+
+/**
+ * Local recovery IT case for changelog. It never fails because local recovery is nice but not
+ * necessary.
+ */
+@RunWith(Parameterized.class)
+public class ChangelogLocalRecoveryITCase {
+
+    private static final int NUM_TASK_MANAGERS = 2;
+    private static final int NUM_TASK_SLOTS = 1;
+
+    @ClassRule public static final TemporaryFolder TEMPORARY_FOLDER = new TemporaryFolder();
+
+    @Parameterized.Parameter public AbstractStateBackend delegatedStateBackend;
+
+    @Parameterized.Parameters(name = "delegated state backend type = {0}")
+    public static Collection<AbstractStateBackend> parameter() {
+        return Arrays.asList(
+                new HashMapStateBackend(),
+                new EmbeddedRocksDBStateBackend(false),
+                new EmbeddedRocksDBStateBackend(true));
+    }
+
+    private MiniClusterWithClientResource cluster;
+    private static String workingDir;
+
+    @BeforeClass
+    public static void setWorkingDir() throws IOException {
+        workingDir = TEMPORARY_FOLDER.newFolder("work").getAbsolutePath();
+    }
+
+    @Before
+    public void setup() throws Exception {
+        Configuration configuration = new Configuration();
+        configuration.setInteger(CheckpointingOptions.MAX_RETAINED_CHECKPOINTS, 1);
+
+        configuration.setString(PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setString(JOB_MANAGER_PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setString(TASK_MANAGER_PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setBoolean(LOCAL_RECOVERY, true);
+        FsStateChangelogStorageFactory.configure(
+                configuration, TEMPORARY_FOLDER.newFolder(), Duration.ofMillis(1000), 1);
+        cluster =
+                new MiniClusterWithClientResource(
+                        new MiniClusterResourceConfiguration.Builder()
+                                .setConfiguration(configuration)
+                                .setNumberTaskManagers(NUM_TASK_MANAGERS)
+                                .setNumberSlotsPerTaskManager(NUM_TASK_SLOTS)
+                                .build());
+        cluster.before();
+        cluster.getMiniCluster().overrideRestoreModeForChangelogStateBackend();
+    }
+
+    private JobGraph buildJobGraph(StreamExecutionEnvironment env) {
+        env.addSource(new InfiniteIntegerSource())
+                .setParallelism(1)
+                .keyBy(element -> element)
+                .process(new CountFunction())
+                .addSink(new CollectionSink())
+                .setParallelism(1);
+        return env.getStreamGraph().getJobGraph();
+    }
+
+    @Test
+    public void testRestartTM() throws Exception {
+        File checkpointFolder = TEMPORARY_FOLDER.newFolder();
+        MiniCluster miniCluster = cluster.getMiniCluster();
+        StreamExecutionEnvironment env1 =
+                getEnv(delegatedStateBackend, checkpointFolder, true, 200, 800);
+        JobGraph firstJobGraph = buildJobGraph(env1);
+
+        miniCluster.submitJob(firstJobGraph).get();
+        waitForAllTaskRunning(miniCluster, firstJobGraph.getJobID(), false);
+        // wait job for doing materialization.
+        Thread.sleep(2000);

Review Comment:
   Could you replace it with the actual check that there's a checkpoint with some materialized state? (some tests are already doing that).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926124747


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))

Review Comment:
   Should we extract this into a field or function already?:) 
   
   (also duplicated in parent)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r930115553


##########
flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java:
##########
@@ -407,8 +408,15 @@ public RunnableFuture<SnapshotResult<KeyedStateHandle>> snapshot(
                                         metrics.reportSnapshotResult(snapshotResult))
                         .thenApply(
                                 snapshotResult ->
-                                        SnapshotResult.of(
-                                                snapshotResult.getJobManagerOwnedSnapshot())));
+                                        snapshotResult.getTaskLocalSnapshot() == null
+                                                        || snapshotResult
+                                                                        .getJobManagerOwnedSnapshot()
+                                                                == null
+                                                ? SnapshotResult.of(
+                                                        snapshotResult.getJobManagerOwnedSnapshot())
+                                                : SnapshotResult.withLocalState(
+                                                        snapshotResult.getJobManagerOwnedSnapshot(),
+                                                        snapshotResult.getTaskLocalSnapshot())));

Review Comment:
   nit: there is no need to re-create the result object, and more importantly this is less readable than just casting:
   ```
   .thenApply(this::castSnapshotResult));
   ...
   @SuppressWarnings("unchecked")
   private SnapshotResult<KeyedStateHandle> castSnapshotResult(SnapshotResult<?> snapshotResult) {
       return (SnapshotResult<KeyedStateHandle>) snapshotResult;
   }
   
   ```



##########
flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java:
##########
@@ -407,8 +408,15 @@ public RunnableFuture<SnapshotResult<KeyedStateHandle>> snapshot(
                                         metrics.reportSnapshotResult(snapshotResult))
                         .thenApply(
                                 snapshotResult ->
-                                        SnapshotResult.of(
-                                                snapshotResult.getJobManagerOwnedSnapshot())));
+                                        snapshotResult.getTaskLocalSnapshot() == null
+                                                        || snapshotResult
+                                                                        .getJobManagerOwnedSnapshot()
+                                                                == null
+                                                ? SnapshotResult.of(
+                                                        snapshotResult.getJobManagerOwnedSnapshot())
+                                                : SnapshotResult.withLocalState(
+                                                        snapshotResult.getJobManagerOwnedSnapshot(),
+                                                        snapshotResult.getTaskLocalSnapshot())));

Review Comment:
   nit: there is no need to re-create the result object, and more importantly this is less readable than just casting:
   ```
   .thenApply(this::castSnapshotResult));
   ...
   @SuppressWarnings("unchecked")
   private SnapshotResult<KeyedStateHandle> castSnapshotResult(SnapshotResult<?> snapshotResult) {
       return (SnapshotResult<KeyedStateHandle>) snapshotResult;
   }
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r920731963


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendLocalHandle.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.changelog;
+
+import org.apache.flink.runtime.state.KeyGroupRange;
+import org.apache.flink.runtime.state.KeyedStateHandle;
+import org.apache.flink.runtime.state.SharedStateRegistry;
+import org.apache.flink.runtime.state.StateHandleID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.List;
+
+public class ChangelogStateBackendLocalHandle implements ChangelogStateBackendHandle {
+    private static final long serialVersionUID = 1L;
+    private static final Logger LOG =
+            LoggerFactory.getLogger(ChangelogStateBackendLocalHandle.class);
+    private final List<KeyedStateHandle> localMaterialized;
+    private final List<ChangelogStateHandle> localNonMaterialized;
+    private final ChangelogStateBackendHandleImpl remoteHandle;
+
+    public ChangelogStateBackendLocalHandle(
+            List<KeyedStateHandle> localMaterialized,
+            List<ChangelogStateHandle> localNonMaterialized,
+            ChangelogStateBackendHandleImpl remoteHandle) {
+        this.localMaterialized = localMaterialized;
+        this.localNonMaterialized = localNonMaterialized;
+        this.remoteHandle = remoteHandle;
+    }
+
+    @Override
+    public List<KeyedStateHandle> getMaterializedStateHandles() {
+        return localMaterialized;
+    }
+
+    @Override
+    public List<ChangelogStateHandle> getNonMaterializedStateHandles() {
+        return localNonMaterialized;
+    }
+
+    @Override
+    public long getMaterializationID() {
+        return remoteHandle.getMaterializationID();
+    }
+
+    @Override
+    public ChangelogStateBackendHandle rebound(long checkpointId) {

Review Comment:
   >  changing config will not triggers local recovery and the rebound method will not be called.
   
   How about throwing `UnsupportedOperationException("Should not call here.")` here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r895055680


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStore.java:
##########
@@ -76,7 +74,7 @@ public interface TaskLocalStateStore {
     /**
      * Remove all checkpoints from the store that match the given predicate.
      *
-     * @param matcher the predicate that selects the checkpoints for pruning.
+     * @param checkpointID the checkpoints for pruning.
      */
-    void pruneMatchingCheckpoints(LongPredicate matcher);
+    void pruneNotMatchingCheckpoints(long checkpointID);

Review Comment:
   This change (and the corresponding call sites) seems like a pure refactoring.
   If so, it's preferrable to put it into a separate commit according to [Flink guidelines](https://flink.apache.org/contributing/code-style-and-quality-pull-requests.html#separate-refactoring-cleanup-and-independent-changes)



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -307,14 +340,23 @@ public void abortCheckpoint(long abortedCheckpointId) {
                 jobVertexID,
                 subtaskIndex);
 
+        // delete the referenced checkpoint meantime
         pruneCheckpoints(
-                snapshotCheckpointId -> snapshotCheckpointId == abortedCheckpointId, false);
+                snapshotCheckpointId ->
+                        (snapshotCheckpointId == abortedCheckpointId
+                                || referredByCheckpointID.getOrDefault(snapshotCheckpointId, -1L)
+                                        == abortedCheckpointId),

Review Comment:
   Do I understand correctly that in case of changelog
   - discard does nothing for jm part, because of the empty 
   `discardState()` in `ChangelogStateBackendHandleImpl`
   - the local part can is discarded, although it could be re-used; but it won't hurt correctness (on recovery, non-local state will be used)
   
   ?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -141,6 +158,29 @@ public void storeLocalState(
                     subtaskIndex);
         }
 
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (currentMaterializationID == null

Review Comment:
   This assumes that the snapshot contains at most one materialization ID.
   Can (and should) we rely on it? 
   I'm not sure whether it holds for down-scaling case.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -141,6 +158,29 @@ public void storeLocalState(
                     subtaskIndex);
         }
 
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (currentMaterializationID == null
+                            || materializationID != currentMaterializationID.f0) {
+                        currentMaterializationID = Tuple2.of(materializationID, checkpointId);
+                        referredByCheckpointID.clear();

Review Comment:
   Could you clarify what happens if two snapshots with different materialization IDs complete out of order?
   Isn't it possible that an older checkpoint (say, CP1 with mID=1) will "override" the newer one (say, CP2 with mID=2)?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -141,6 +158,29 @@ public void storeLocalState(
                     subtaskIndex);
         }
 
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (currentMaterializationID == null
+                            || materializationID != currentMaterializationID.f0) {
+                        currentMaterializationID = Tuple2.of(materializationID, checkpointId);
+                        referredByCheckpointID.clear();
+                    } else {
+                        referredByCheckpointID.put(currentMaterializationID.f1, checkpointId);

Review Comment:
   Shouldn't this (and other access) be synchronized, similar to `storedTaskStateByCheckpointID`?
   
   The method is called from `AsyncCheckpointRunnable`, which can be executed by different threads (plus other methods).



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -98,6 +104,15 @@ public class TaskLocalStateStoreImpl implements OwnedTaskLocalStateStore {
     @GuardedBy("lock")
     private final SortedMap<Long, TaskStateSnapshot> storedTaskStateByCheckpointID;
 
+    /** The relationship between checkpoints. (cp1,cp3) means cp3 refer to cp1's state. */
+    private final Map<Long, Long> referredByCheckpointID;
+
+    /**
+     * (MaterializationID, created by checkpointID). Because local store only keeps one checkpoint,
+     * at most one materialization can exist at the same time.
+     */
+    private Tuple2<Long, Long> currentMaterializationID;

Review Comment:
   `lastChangelogMaterializationIdAndCheckpoint`?
   
   Or maybe these two new fields should form a new helper class encapsulating changelog-specific logic? 
   And then probably tuple2 can be replaced with just two fields?
   
   I think it should be more clear and easier to test. WDYT?



##########
flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java:
##########
@@ -804,6 +833,9 @@ private static class ChangelogSnapshotState {
          */
         private final List<KeyedStateHandle> materializedSnapshot;
 
+        /** Materialized snapshot from the underlying delegated state backend for local recovery. */
+        private final List<KeyedStateHandle> localMaterializedSnapshot;

Review Comment:
   There is already `SnapshotResult` which is responsible for holding and distinguishing local and jm-owned snapshots. With this change, `ChangelogSnapshotState` overlaps with it.
   
   Maybe it makes sense to refactor`ChangelogSnapshotState` so that it holds a single `SnapshotResult`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] masteryhx commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
masteryhx commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r895363701


##########
flink-runtime/src/test/java/org/apache/flink/runtime/state/TaskLocalStateStoreImplTest.java:
##########
@@ -180,6 +216,19 @@ public void abortCheckpoint() throws Exception {
         checkStoredAsExpected(taskStateSnapshots, aborted + 1, chkCount);
     }
 
+    public void abortCheckpointForChangelog() throws Exception {

Review Comment:
   Maybe lost the annotation of "@Test".



##########
flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java:
##########
@@ -407,8 +407,9 @@ public RunnableFuture<SnapshotResult<KeyedStateHandle>> snapshot(
                                         metrics.reportSnapshotResult(snapshotResult))
                         .thenApply(
                                 snapshotResult ->
-                                        SnapshotResult.of(
-                                                snapshotResult.getJobManagerOwnedSnapshot())));
+                                        SnapshotResult.withLocalState(

Review Comment:
   There are "@Nonnull" annotations in SnapshotResult#withLocalState.
   So I think it's better to adding some check before calling it.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -98,6 +104,15 @@ public class TaskLocalStateStoreImpl implements OwnedTaskLocalStateStore {
     @GuardedBy("lock")
     private final SortedMap<Long, TaskStateSnapshot> storedTaskStateByCheckpointID;
 
+    /** The relationship between checkpoints. (cp1,cp3) means cp3 refer to cp1's state. */
+    private final Map<Long, Long> referredByCheckpointID;
+
+    /**
+     * (MaterializationID, created by checkpointID). Because local store only keeps one checkpoint,
+     * at most one materialization can exist at the same time.
+     */
+    private Tuple2<Long, Long> currentMaterializationID;
+
     public TaskLocalStateStoreImpl(

Review Comment:
   Maybe considering to extract some codes related to changelog to a new subclass?



##########
flink-tests/src/test/java/org/apache/flink/test/checkpointing/ChangelogLocalRecoveryITCase.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.checkpointing;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.changelog.fs.FsStateChangelogStorageFactory;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.StateChangelogOptions;
+import org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.minicluster.MiniCluster;
+import org.apache.flink.runtime.state.AbstractStateBackend;
+import org.apache.flink.runtime.state.StateBackend;
+import org.apache.flink.runtime.state.hashmap.HashMapStateBackend;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CollectionSink;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CountFunction;
+import org.apache.flink.test.util.InfiniteIntegerSource;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.File;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.configuration.CheckpointingOptions.LOCAL_RECOVERY;
+import static org.apache.flink.configuration.ClusterOptions.JOB_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.TASK_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning;
+
+/**
+ * Local recovery IT case for changelog. It never fails because local recovery is nice but not
+ * necessary.
+ */
+@RunWith(Parameterized.class)
+public class ChangelogLocalRecoveryITCase {

Review Comment:
   If the procedure of local recovery has some issues, the ITCase may still retain success ?



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -213,24 +253,14 @@ public TaskStateSnapshot retrieveLocalState(long checkpointID) {
         }
 
         if (snapshot != null) {
-            if (LOG.isTraceEnabled()) {
-                LOG.trace(
-                        "Found registered local state for checkpoint {} in subtask ({} - {} - {}) : {}",
-                        checkpointID,
-                        jobID,
-                        jobVertexID,
-                        subtaskIndex,
-                        snapshot);
-            } else if (LOG.isDebugEnabled()) {
-                LOG.debug(
-                        "Found registered local state for checkpoint {} in subtask ({} - {} - {})",
-                        checkpointID,
-                        jobID,
-                        jobVertexID,
-                        subtaskIndex);
-            }
+            LOG.info(
+                    "Found registered local state for checkpoint {} in subtask ({} - {} - {})",

Review Comment:
   Why removing logging "snapshot" ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r895935387


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -98,6 +104,15 @@ public class TaskLocalStateStoreImpl implements OwnedTaskLocalStateStore {
     @GuardedBy("lock")
     private final SortedMap<Long, TaskStateSnapshot> storedTaskStateByCheckpointID;
 
+    /** The relationship between checkpoints. (cp1,cp3) means cp3 refer to cp1's state. */
+    private final Map<Long, Long> referredByCheckpointID;
+
+    /**
+     * (MaterializationID, created by checkpointID). Because local store only keeps one checkpoint,
+     * at most one materialization can exist at the same time.
+     */
+    private Tuple2<Long, Long> currentMaterializationID;
+
     public TaskLocalStateStoreImpl(

Review Comment:
   Nice suggestion, I created ChangelogTaskLocalStateStore to encapsulate changelog-specific logic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926125810


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))
+                        .subtaskBaseDirectory(checkpointId);
+        File directoryForChangelog =
+                new File(checkpointDirectory, CHANGE_LOG_CHECKPOINT_PREFIX + checkpointId);
+
+        if (!directoryForChangelog.exists() && !directoryForChangelog.mkdirs()) {
+            throw new FlinkRuntimeException(
+                    String.format(
+                            "Could not create the checkpoint directory '%s'",
+                            directoryForChangelog));
+        }
+
+        return directoryForChangelog;
+    }
+
+    private void deleteMaterialization(LongPredicate pruningChecker) {
+        final Set<Long> materializationToRemove = new HashSet<>();
+        synchronized (lock) {
+            Iterator<Entry<Long, Long>> iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                long entryCheckpointId = entry.getKey();
+                if (pruningChecker.test(entryCheckpointId)) {
+                    materializationToRemove.add(entry.getValue());
+                    iterator.remove();
+                }
+            }
+
+            iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                materializationToRemove.remove(entry.getValue());
+            }

Review Comment:
   How about simplifying this code?
   ```
   Set<Long> checkpoints =
           mapToMaterializationId.keySet().stream()
                   .filter(pruningChecker::test)
                   .collect(Collectors.toSet());
   materializationToRemove =
           checkpoints.stream()
                   .map(mapToMaterializationId::remove)
                   .collect(Collectors.toSet());
   materializationToRemove.removeAll(mapToMaterializationId.values());
   ```
   
   Alternatively, we could maintain a `Map<materializationId, Set<checkpointId>>` and discard materialization whenever the set is empty. But two maps seems more complex.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926126113


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))
+                        .subtaskBaseDirectory(checkpointId);
+        File directoryForChangelog =
+                new File(checkpointDirectory, CHANGE_LOG_CHECKPOINT_PREFIX + checkpointId);
+
+        if (!directoryForChangelog.exists() && !directoryForChangelog.mkdirs()) {
+            throw new FlinkRuntimeException(
+                    String.format(
+                            "Could not create the checkpoint directory '%s'",
+                            directoryForChangelog));
+        }
+
+        return directoryForChangelog;
+    }
+
+    private void deleteMaterialization(LongPredicate pruningChecker) {
+        final Set<Long> materializationToRemove = new HashSet<>();
+        synchronized (lock) {
+            Iterator<Entry<Long, Long>> iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                long entryCheckpointId = entry.getKey();
+                if (pruningChecker.test(entryCheckpointId)) {
+                    materializationToRemove.add(entry.getValue());
+                    iterator.remove();
+                }
+            }
+
+            iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                materializationToRemove.remove(entry.getValue());
+            }
+        }
+
+        for (Long materializationId : materializationToRemove) {
+            File materializedDir =
+                    localRecoveryConfig
+                            .getLocalStateDirectoryProvider()
+                            .orElseThrow(
+                                    () ->
+                                            new IllegalStateException(
+                                                    "Local recovery must be enabled."))
+                            .subtaskSpecificCheckpointDirectory(materializationId);
+
+            if (!materializedDir.exists()) {
+                continue;
+            }

Review Comment:
   nit: invert `if`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r926352380


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/ChangelogTaskLocalStateStore.java:
##########
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.runtime.checkpoint.OperatorSubtaskState;
+import org.apache.flink.runtime.checkpoint.TaskStateSnapshot;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.OperatorID;
+import org.apache.flink.runtime.state.changelog.ChangelogStateBackendHandle;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executor;
+import java.util.function.LongPredicate;
+
+/** Changelog's implementation of a {@link TaskLocalStateStore}. */
+public class ChangelogTaskLocalStateStore extends TaskLocalStateStoreImpl {
+
+    private static final Logger LOG = LoggerFactory.getLogger(ChangelogTaskLocalStateStore.class);
+
+    private static final String CHANGE_LOG_CHECKPOINT_PREFIX = "changelog_chk_";
+
+    /**
+     * The mapper of checkpointId and materializationId. (cp3, materializationId2) means cp3 refer
+     * to m1.
+     */
+    private final Map<Long, Long> mapToMaterializationId;
+
+    /** Last checkpointId, to check whether checkpoint is out of order. */
+    private long lastCheckpointId = -1L;
+
+    public ChangelogTaskLocalStateStore(
+            @Nonnull JobID jobID,
+            @Nonnull AllocationID allocationID,
+            @Nonnull JobVertexID jobVertexID,
+            @Nonnegative int subtaskIndex,
+            @Nonnull LocalRecoveryConfig localRecoveryConfig,
+            @Nonnull Executor discardExecutor) {
+        super(jobID, allocationID, jobVertexID, subtaskIndex, localRecoveryConfig, discardExecutor);
+        this.mapToMaterializationId = new HashMap<>();
+    }
+
+    private void updateReference(long checkpointId, TaskStateSnapshot localState) {
+        if (localState == null) {
+            localState = NULL_DUMMY;
+        }
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (mapToMaterializationId.getOrDefault(checkpointId, Long.MAX_VALUE)
+                            < materializationID) {
+                        LOG.info(
+                                "Update checkpoint {}, old materializationID {}, new materializationID {}.",
+                                checkpointId,
+                                mapToMaterializationId.get(checkpointId),
+                                materializationID);
+                    }
+                    mapToMaterializationId.put(checkpointId, materializationID);
+                }
+            }
+        }
+    }
+
+    @Override
+    public void storeLocalState(long checkpointId, @Nullable TaskStateSnapshot localState) {
+        if (checkpointId < lastCheckpointId) {
+            LOG.info(
+                    "Current checkpoint {} is out of order, smaller than last CheckpointId {}.",
+                    lastCheckpointId,
+                    checkpointId);
+            return;
+        } else {
+            lastCheckpointId = checkpointId;
+        }
+        synchronized (lock) {
+            updateReference(checkpointId, localState);
+        }
+        super.storeLocalState(checkpointId, localState);
+    }
+
+    @Override
+    protected File getCheckpointDirectory(long checkpointId) {
+        final File checkpointDirectory =
+                localRecoveryConfig
+                        .getLocalStateDirectoryProvider()
+                        .orElseThrow(
+                                () -> new IllegalStateException("Local recovery must be enabled."))
+                        .subtaskBaseDirectory(checkpointId);
+        File directoryForChangelog =
+                new File(checkpointDirectory, CHANGE_LOG_CHECKPOINT_PREFIX + checkpointId);
+
+        if (!directoryForChangelog.exists() && !directoryForChangelog.mkdirs()) {
+            throw new FlinkRuntimeException(
+                    String.format(
+                            "Could not create the checkpoint directory '%s'",
+                            directoryForChangelog));
+        }
+
+        return directoryForChangelog;
+    }
+
+    private void deleteMaterialization(LongPredicate pruningChecker) {
+        final Set<Long> materializationToRemove = new HashSet<>();
+        synchronized (lock) {
+            Iterator<Entry<Long, Long>> iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                long entryCheckpointId = entry.getKey();
+                if (pruningChecker.test(entryCheckpointId)) {
+                    materializationToRemove.add(entry.getValue());
+                    iterator.remove();
+                }
+            }
+
+            iterator = mapToMaterializationId.entrySet().iterator();
+            while (iterator.hasNext()) {
+                Map.Entry<Long, Long> entry = iterator.next();
+                materializationToRemove.remove(entry.getValue());
+            }
+        }
+
+        for (Long materializationId : materializationToRemove) {
+            File materializedDir =
+                    localRecoveryConfig
+                            .getLocalStateDirectoryProvider()
+                            .orElseThrow(
+                                    () ->
+                                            new IllegalStateException(
+                                                    "Local recovery must be enabled."))
+                            .subtaskSpecificCheckpointDirectory(materializationId);
+
+            if (!materializedDir.exists()) {
+                continue;
+            }
+            try {
+                deleteDirectory(materializedDir);

Review Comment:
   - For RocksDB(incr=false) and heap, they only use `chk_` for local state.
   - For RocksDB(incr=true), `IncrementalLocalKeyedStateHandle` extends `DirectoryKeyedStateHandle`, all SST files would be placed in `DirectoryKeyedStateHandle`, no shared state. So, it also only uses `chk_` for local state.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] rkhachatryan commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r928324559


##########
flink-tests/src/test/java/org/apache/flink/test/checkpointing/ChangelogLocalRecoveryITCase.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.checkpointing;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.changelog.fs.FsStateChangelogStorageFactory;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.StateChangelogOptions;
+import org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.minicluster.MiniCluster;
+import org.apache.flink.runtime.state.AbstractStateBackend;
+import org.apache.flink.runtime.state.StateBackend;
+import org.apache.flink.runtime.state.hashmap.HashMapStateBackend;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CollectionSink;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CountFunction;
+import org.apache.flink.test.util.InfiniteIntegerSource;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.File;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.configuration.CheckpointingOptions.LOCAL_RECOVERY;
+import static org.apache.flink.configuration.ClusterOptions.JOB_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.TASK_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning;
+
+/**
+ * Local recovery IT case for changelog. It never fails because local recovery is nice but not
+ * necessary.
+ */
+@RunWith(Parameterized.class)
+public class ChangelogLocalRecoveryITCase {
+
+    private static final int NUM_TASK_MANAGERS = 2;
+    private static final int NUM_TASK_SLOTS = 1;
+
+    @ClassRule public static final TemporaryFolder TEMPORARY_FOLDER = new TemporaryFolder();
+
+    @Parameterized.Parameter public AbstractStateBackend delegatedStateBackend;
+
+    @Parameterized.Parameters(name = "delegated state backend type = {0}")
+    public static Collection<AbstractStateBackend> parameter() {
+        return Arrays.asList(
+                new HashMapStateBackend(),
+                new EmbeddedRocksDBStateBackend(false),
+                new EmbeddedRocksDBStateBackend(true));
+    }
+
+    private MiniClusterWithClientResource cluster;
+    private static String workingDir;
+
+    @BeforeClass
+    public static void setWorkingDir() throws IOException {
+        workingDir = TEMPORARY_FOLDER.newFolder("work").getAbsolutePath();
+    }
+
+    @Before
+    public void setup() throws Exception {
+        Configuration configuration = new Configuration();
+        configuration.setInteger(CheckpointingOptions.MAX_RETAINED_CHECKPOINTS, 1);
+
+        configuration.setString(PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setString(JOB_MANAGER_PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setString(TASK_MANAGER_PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setBoolean(LOCAL_RECOVERY, true);
+        FsStateChangelogStorageFactory.configure(
+                configuration, TEMPORARY_FOLDER.newFolder(), Duration.ofMillis(1000), 1);
+        cluster =
+                new MiniClusterWithClientResource(
+                        new MiniClusterResourceConfiguration.Builder()
+                                .setConfiguration(configuration)
+                                .setNumberTaskManagers(NUM_TASK_MANAGERS)
+                                .setNumberSlotsPerTaskManager(NUM_TASK_SLOTS)
+                                .build());
+        cluster.before();
+        cluster.getMiniCluster().overrideRestoreModeForChangelogStateBackend();
+    }
+
+    private JobGraph buildJobGraph(StreamExecutionEnvironment env) {
+        env.addSource(new InfiniteIntegerSource())
+                .setParallelism(1)
+                .keyBy(element -> element)
+                .process(new CountFunction())
+                .addSink(new CollectionSink())
+                .setParallelism(1);
+        return env.getStreamGraph().getJobGraph();
+    }
+
+    @Test
+    public void testRestartTM() throws Exception {
+        File checkpointFolder = TEMPORARY_FOLDER.newFolder();
+        MiniCluster miniCluster = cluster.getMiniCluster();
+        StreamExecutionEnvironment env1 =
+                getEnv(delegatedStateBackend, checkpointFolder, true, 200, 800);
+        JobGraph firstJobGraph = buildJobGraph(env1);
+
+        miniCluster.submitJob(firstJobGraph).get();
+        waitForAllTaskRunning(miniCluster, firstJobGraph.getJobID(), false);
+        // wait job for doing materialization.
+        Thread.sleep(2000);

Review Comment:
   `ChangelogLocalRecoveryITCase` doesn't actually test materialization, so it shouldn't extend `ChangelogPeriodicMaterializationTestBase`, should it?
   
   WDYT about reusing the code in some different way (`static`?) or renaming/restructuring the base class?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r928364178


##########
flink-tests/src/test/java/org/apache/flink/test/checkpointing/ChangelogLocalRecoveryITCase.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.checkpointing;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.changelog.fs.FsStateChangelogStorageFactory;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.StateChangelogOptions;
+import org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.minicluster.MiniCluster;
+import org.apache.flink.runtime.state.AbstractStateBackend;
+import org.apache.flink.runtime.state.StateBackend;
+import org.apache.flink.runtime.state.hashmap.HashMapStateBackend;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CollectionSink;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CountFunction;
+import org.apache.flink.test.util.InfiniteIntegerSource;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.File;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.configuration.CheckpointingOptions.LOCAL_RECOVERY;
+import static org.apache.flink.configuration.ClusterOptions.JOB_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.TASK_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning;
+
+/**
+ * Local recovery IT case for changelog. It never fails because local recovery is nice but not
+ * necessary.
+ */
+@RunWith(Parameterized.class)
+public class ChangelogLocalRecoveryITCase {
+
+    private static final int NUM_TASK_MANAGERS = 2;
+    private static final int NUM_TASK_SLOTS = 1;
+
+    @ClassRule public static final TemporaryFolder TEMPORARY_FOLDER = new TemporaryFolder();
+
+    @Parameterized.Parameter public AbstractStateBackend delegatedStateBackend;
+
+    @Parameterized.Parameters(name = "delegated state backend type = {0}")
+    public static Collection<AbstractStateBackend> parameter() {
+        return Arrays.asList(
+                new HashMapStateBackend(),
+                new EmbeddedRocksDBStateBackend(false),
+                new EmbeddedRocksDBStateBackend(true));
+    }
+
+    private MiniClusterWithClientResource cluster;
+    private static String workingDir;
+
+    @BeforeClass
+    public static void setWorkingDir() throws IOException {
+        workingDir = TEMPORARY_FOLDER.newFolder("work").getAbsolutePath();
+    }
+
+    @Before
+    public void setup() throws Exception {
+        Configuration configuration = new Configuration();
+        configuration.setInteger(CheckpointingOptions.MAX_RETAINED_CHECKPOINTS, 1);
+
+        configuration.setString(PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setString(JOB_MANAGER_PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setString(TASK_MANAGER_PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setBoolean(LOCAL_RECOVERY, true);
+        FsStateChangelogStorageFactory.configure(
+                configuration, TEMPORARY_FOLDER.newFolder(), Duration.ofMillis(1000), 1);
+        cluster =
+                new MiniClusterWithClientResource(
+                        new MiniClusterResourceConfiguration.Builder()
+                                .setConfiguration(configuration)
+                                .setNumberTaskManagers(NUM_TASK_MANAGERS)
+                                .setNumberSlotsPerTaskManager(NUM_TASK_SLOTS)
+                                .build());
+        cluster.before();
+        cluster.getMiniCluster().overrideRestoreModeForChangelogStateBackend();
+    }
+
+    private JobGraph buildJobGraph(StreamExecutionEnvironment env) {
+        env.addSource(new InfiniteIntegerSource())
+                .setParallelism(1)
+                .keyBy(element -> element)
+                .process(new CountFunction())
+                .addSink(new CollectionSink())
+                .setParallelism(1);
+        return env.getStreamGraph().getJobGraph();
+    }
+
+    @Test
+    public void testRestartTM() throws Exception {
+        File checkpointFolder = TEMPORARY_FOLDER.newFolder();
+        MiniCluster miniCluster = cluster.getMiniCluster();
+        StreamExecutionEnvironment env1 =
+                getEnv(delegatedStateBackend, checkpointFolder, true, 200, 800);
+        JobGraph firstJobGraph = buildJobGraph(env1);
+
+        miniCluster.submitJob(firstJobGraph).get();
+        waitForAllTaskRunning(miniCluster, firstJobGraph.getJobID(), false);
+        // wait job for doing materialization.
+        Thread.sleep(2000);

Review Comment:
   Resued this code in `static` way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r927677335


##########
flink-tests/src/test/java/org/apache/flink/test/checkpointing/ChangelogLocalRecoveryITCase.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.checkpointing;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.changelog.fs.FsStateChangelogStorageFactory;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.StateChangelogOptions;
+import org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.minicluster.MiniCluster;
+import org.apache.flink.runtime.state.AbstractStateBackend;
+import org.apache.flink.runtime.state.StateBackend;
+import org.apache.flink.runtime.state.hashmap.HashMapStateBackend;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CollectionSink;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CountFunction;
+import org.apache.flink.test.util.InfiniteIntegerSource;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.File;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.configuration.CheckpointingOptions.LOCAL_RECOVERY;
+import static org.apache.flink.configuration.ClusterOptions.JOB_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.TASK_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning;
+
+/**
+ * Local recovery IT case for changelog. It never fails because local recovery is nice but not
+ * necessary.
+ */
+@RunWith(Parameterized.class)
+public class ChangelogLocalRecoveryITCase {
+
+    private static final int NUM_TASK_MANAGERS = 2;
+    private static final int NUM_TASK_SLOTS = 1;
+
+    @ClassRule public static final TemporaryFolder TEMPORARY_FOLDER = new TemporaryFolder();
+
+    @Parameterized.Parameter public AbstractStateBackend delegatedStateBackend;
+
+    @Parameterized.Parameters(name = "delegated state backend type = {0}")
+    public static Collection<AbstractStateBackend> parameter() {
+        return Arrays.asList(
+                new HashMapStateBackend(),
+                new EmbeddedRocksDBStateBackend(false),
+                new EmbeddedRocksDBStateBackend(true));
+    }
+
+    private MiniClusterWithClientResource cluster;
+    private static String workingDir;
+
+    @BeforeClass
+    public static void setWorkingDir() throws IOException {
+        workingDir = TEMPORARY_FOLDER.newFolder("work").getAbsolutePath();
+    }
+
+    @Before
+    public void setup() throws Exception {
+        Configuration configuration = new Configuration();
+        configuration.setInteger(CheckpointingOptions.MAX_RETAINED_CHECKPOINTS, 1);
+
+        configuration.setString(PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setString(JOB_MANAGER_PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setString(TASK_MANAGER_PROCESS_WORKING_DIR_BASE, workingDir);
+        configuration.setBoolean(LOCAL_RECOVERY, true);
+        FsStateChangelogStorageFactory.configure(
+                configuration, TEMPORARY_FOLDER.newFolder(), Duration.ofMillis(1000), 1);
+        cluster =
+                new MiniClusterWithClientResource(
+                        new MiniClusterResourceConfiguration.Builder()
+                                .setConfiguration(configuration)
+                                .setNumberTaskManagers(NUM_TASK_MANAGERS)
+                                .setNumberSlotsPerTaskManager(NUM_TASK_SLOTS)
+                                .build());
+        cluster.before();
+        cluster.getMiniCluster().overrideRestoreModeForChangelogStateBackend();
+    }
+
+    private JobGraph buildJobGraph(StreamExecutionEnvironment env) {
+        env.addSource(new InfiniteIntegerSource())
+                .setParallelism(1)
+                .keyBy(element -> element)
+                .process(new CountFunction())
+                .addSink(new CollectionSink())
+                .setParallelism(1);
+        return env.getStreamGraph().getJobGraph();
+    }
+
+    @Test
+    public void testRestartTM() throws Exception {
+        File checkpointFolder = TEMPORARY_FOLDER.newFolder();
+        MiniCluster miniCluster = cluster.getMiniCluster();
+        StreamExecutionEnvironment env1 =
+                getEnv(delegatedStateBackend, checkpointFolder, true, 200, 800);
+        JobGraph firstJobGraph = buildJobGraph(env1);
+
+        miniCluster.submitJob(firstJobGraph).get();
+        waitForAllTaskRunning(miniCluster, firstJobGraph.getJobID(), false);
+        // wait job for doing materialization.
+        Thread.sleep(2000);

Review Comment:
   👍, I have make `ChangelogLocalRecoveryITCase` extend `ChangelogPeriodicMaterializationTestBase`, and reuse `getAllStateHandleId()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r895423197


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -307,14 +340,23 @@ public void abortCheckpoint(long abortedCheckpointId) {
                 jobVertexID,
                 subtaskIndex);
 
+        // delete the referenced checkpoint meantime
         pruneCheckpoints(
-                snapshotCheckpointId -> snapshotCheckpointId == abortedCheckpointId, false);
+                snapshotCheckpointId ->
+                        (snapshotCheckpointId == abortedCheckpointId
+                                || referredByCheckpointID.getOrDefault(snapshotCheckpointId, -1L)
+                                        == abortedCheckpointId),

Review Comment:
   > 1. discard does nothing for jm part, because of the empty
   discardState() in ChangelogStateBackendHandleImpl
   
   Partially agree:
   - For local state: discard does nothing for jm part, because of the empty
   `discardState()` in `ChangelogStateBackendHandleImpl` and **no local handle would be registered in** `registerSharedStates(SharedStateRegistry stateRegistry, long checkpointID)`
   - For remote state: `SharedStateRegistry` on JM would discard changelogStateHandle.
   
   > 2. the local part can is discarded, although it could be re-used; but it won't hurt correctness (on recovery, non-local state will be used)
   
   Yes, local recovery is nice to have but not a must. `BackendRestorerProcedure#createAndRestore()` would try to restore from remote. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r895525160


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -141,6 +158,29 @@ public void storeLocalState(
                     subtaskIndex);
         }
 
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (currentMaterializationID == null

Review Comment:
   1. Local recovery and rescaling can not happen at the same time, we can temporarily assume that the snapshot contains at most one materialization ID now.
   2. Even if local recovery and rescaling can happen at the same time later,  `storeLocalState()` is called after   restoring is complete, and there is also at most one materialization ID per `ChangelogStateBackendHandle`, because we choose max materializationID to handle rescaling.
   https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java#L616-L617



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r895926048


##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -141,6 +158,29 @@ public void storeLocalState(
                     subtaskIndex);
         }
 
+        for (Map.Entry<OperatorID, OperatorSubtaskState> subtaskStateEntry :
+                localState.getSubtaskStateMappings()) {
+            for (KeyedStateHandle keyedStateHandle :
+                    subtaskStateEntry.getValue().getManagedKeyedState()) {
+                if (keyedStateHandle instanceof ChangelogStateBackendHandle) {
+                    ChangelogStateBackendHandle changelogStateBackendHandle =
+                            (ChangelogStateBackendHandle) keyedStateHandle;
+                    long materializationID = changelogStateBackendHandle.getMaterializationID();
+                    if (currentMaterializationID == null
+                            || materializationID != currentMaterializationID.f0) {
+                        currentMaterializationID = Tuple2.of(materializationID, checkpointId);
+                        referredByCheckpointID.clear();
+                    } else {
+                        referredByCheckpointID.put(currentMaterializationID.f1, checkpointId);

Review Comment:
   Yes, move it into `synchronized` block.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [flink] fredia commented on a diff in pull request #19907: [FLINK-27692][state] Support local recovery for materialized part of changelog

Posted by GitBox <gi...@apache.org>.
fredia commented on code in PR #19907:
URL: https://github.com/apache/flink/pull/19907#discussion_r895935127


##########
flink-tests/src/test/java/org/apache/flink/test/checkpointing/ChangelogLocalRecoveryITCase.java:
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.checkpointing;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.changelog.fs.FsStateChangelogStorageFactory;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.StateChangelogOptions;
+import org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.minicluster.MiniCluster;
+import org.apache.flink.runtime.state.AbstractStateBackend;
+import org.apache.flink.runtime.state.StateBackend;
+import org.apache.flink.runtime.state.hashmap.HashMapStateBackend;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.environment.CheckpointConfig;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CollectionSink;
+import org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationTestBase.CountFunction;
+import org.apache.flink.test.util.InfiniteIntegerSource;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.File;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+
+import static org.apache.flink.configuration.CheckpointingOptions.LOCAL_RECOVERY;
+import static org.apache.flink.configuration.ClusterOptions.JOB_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.configuration.ClusterOptions.TASK_MANAGER_PROCESS_WORKING_DIR_BASE;
+import static org.apache.flink.runtime.testutils.CommonTestUtils.waitForAllTaskRunning;
+
+/**
+ * Local recovery IT case for changelog. It never fails because local recovery is nice but not
+ * necessary.
+ */
+@RunWith(Parameterized.class)
+public class ChangelogLocalRecoveryITCase {

Review Comment:
   Yes, It is.



##########
flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskLocalStateStoreImpl.java:
##########
@@ -213,24 +253,14 @@ public TaskStateSnapshot retrieveLocalState(long checkpointID) {
         }
 
         if (snapshot != null) {
-            if (LOG.isTraceEnabled()) {
-                LOG.trace(
-                        "Found registered local state for checkpoint {} in subtask ({} - {} - {}) : {}",
-                        checkpointID,
-                        jobID,
-                        jobVertexID,
-                        subtaskIndex,
-                        snapshot);
-            } else if (LOG.isDebugEnabled()) {
-                LOG.debug(
-                        "Found registered local state for checkpoint {} in subtask ({} - {} - {})",
-                        checkpointID,
-                        jobID,
-                        jobVertexID,
-                        subtaskIndex);
-            }
+            LOG.info(
+                    "Found registered local state for checkpoint {} in subtask ({} - {} - {})",

Review Comment:
   Because "snapshot"  may be `NULL_DUMMY`.  I reverted it back.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org