You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/10/22 00:09:46 UTC

[GitHub] [flink] AHeise opened a new pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

AHeise opened a new pull request #13735:
URL: https://github.com/apache/flink/pull/13735


   <!--
   *Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
   
   *Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
   
   ## Contribution Checklist
   
     - Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
     
     - Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
     Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
     
     - Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
   
     - Each pull request should address only one issue, not mix up code from multiple issues.
     
     - Each commit in the pull request has a meaningful commit message (including the JIRA id)
   
     - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
   
   
   **(The sections below can be removed for hotfixes of typos)**
   -->
   
   ## What is the purpose of the change
   
   Currently, recovery with rescaled channel state is failing. This PR enhances `StateAssignmentOperation` to reassignment channel state and create a channel state mapping, where channels before rescaling are mapped to channels after rescaling.
   
   ## Brief change log
   
   - Refactors `StateAssignmentOperation` in preparation. In particular, it adds a two pass approach such that vertex may assign state of other vertexes.
   - Adds a method for `StreamPartitioner` to determine the affected partitions during rescale operations. (Added to `ChannelSelector` because of separation of `runtime` and `streaming`)
   - Add `StreamPartitioner` to `JobEdge`, such that it is assessible to `StateAssignmentOperation`.
   - Adding channel reassignment to `StateAssignmentOperation`.
   
   
   ## Verifying this change
   
   Added unit tests. The final rescale PRs will also adapt `UnalignedCheckpointITCase` to cover rescaling.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / **no**)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no)
     - The serializers: (yes / **no** / don't know)
     - The runtime per-record code paths (performance sensitive): (**yes** / no / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
     - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (**yes** / no)
     - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) (not finished yet)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 470d466cfe4455aefd5a302cedd92b57750f2c57 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517348994



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -327,4 +289,252 @@ public boolean hasState() {
 			|| inputChannelState.hasState()
 			|| resultSubpartitionState.hasState();
 	}
+
+	public static Builder builder() {
+		return new Builder();
+	}
+
+	/**
+	 * The builder for a new {@link OperatorSubtaskState} which can be obtained by {@link #builder()}.
+	 */
+	public static class Builder {
+		private StateObjectCollection<OperatorStateHandle> managedOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<OperatorStateHandle> rawOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> managedKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> rawKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<InputChannelStateHandle> inputChannelState = StateObjectCollection.empty();
+		private StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState = StateObjectCollection.empty();
+		private VirtualChannelMapping inputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+		private VirtualChannelMapping outputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+
+		private Builder() {
+		}
+
+		public Builder setManagedOperatorState(StateObjectCollection<OperatorStateHandle> managedOperatorState) {
+			this.managedOperatorState = checkNotNull(managedOperatorState);
+			return this;
+		}
+
+		public Builder setManagedOperatorState(OperatorStateHandle managedOperatorState) {
+			return setManagedOperatorState(StateObjectCollection.singleton(checkNotNull(managedOperatorState)));
+		}
+
+		public Builder setRawOperatorState(StateObjectCollection<OperatorStateHandle> rawOperatorState) {
+			this.rawOperatorState = checkNotNull(rawOperatorState);
+			return this;
+		}
+
+		public Builder setRawOperatorState(OperatorStateHandle rawOperatorState) {
+			return setRawOperatorState(StateObjectCollection.singleton(checkNotNull(rawOperatorState)));
+		}
+
+		public Builder setManagedKeyedState(StateObjectCollection<KeyedStateHandle> managedKeyedState) {
+			this.managedKeyedState = checkNotNull(managedKeyedState);
+			return this;
+		}
+
+		public Builder setManagedKeyedState(KeyedStateHandle managedKeyedState) {
+			return setManagedKeyedState(StateObjectCollection.singleton(checkNotNull(managedKeyedState)));
+		}
+
+		public Builder setRawKeyedState(StateObjectCollection<KeyedStateHandle> rawKeyedState) {
+			this.rawKeyedState = checkNotNull(rawKeyedState);
+			return this;
+		}
+
+		public Builder setRawKeyedState(KeyedStateHandle rawKeyedState) {
+			return setRawKeyedState(StateObjectCollection.singleton(checkNotNull(rawKeyedState)));
+		}
+
+		public Builder setInputChannelState(StateObjectCollection<InputChannelStateHandle> inputChannelState) {
+			this.inputChannelState = checkNotNull(inputChannelState);
+			return this;
+		}
+
+		public Builder setResultSubpartitionState(StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState) {
+			this.resultSubpartitionState = checkNotNull(resultSubpartitionState);
+			return this;
+		}
+
+		public Builder setInputChannelMappings(VirtualChannelMapping inputChannelMappings) {
+			this.inputChannelMappings = checkNotNull(inputChannelMappings);
+			return this;
+		}
+
+		public Builder setOutputChannelMappings(VirtualChannelMapping outputChannelMappings) {
+			this.outputChannelMappings = checkNotNull(outputChannelMappings);
+			return this;
+		}
+
+		public OperatorSubtaskState build() {
+			return new OperatorSubtaskState(
+				managedOperatorState,
+				rawOperatorState,
+				managedKeyedState,
+				rawKeyedState,
+				inputChannelState,
+				resultSubpartitionState,
+				inputChannelMappings,
+				outputChannelMappings);
+		}
+	}
+
+	/**
+	 * Captures ambiguous mappings of old channels to new channels.
+	 *
+	 * <p>For inputs, this mapping implies the following:
+	 * <li>
+	 *     <ul>{@link #oldTaskInstances} is set when there is a rescale on this task potentially leading to different
+	 *     key groups. Upstream task has a corresponding {@link #partitionMappings} where it sends data over
+	 *     virtual channel while specifying the channel index in the VirtualChannelSelector. This subtask then
+	 *     demultiplexes over the virtual subtask index.</ul>
+	 *     <ul>{@link #partitionMappings} is set when there is a downscale of the upstream task. Upstream task has
+	 *     a corresponding {@link #oldTaskInstances} where it sends data over virtual channel while specifying the
+	 *     subtask index in the VirtualChannelSelector. This subtask then demultiplexes over channel indexes.</ul>
+	 * </li>
+	 *
+	 * <p>For outputs, it's vice-versa. The information must be kept in sync but they are used in opposite ways for
+	 * multiplexing/demultiplexing.
+	 *
+	 * <p>Note that in the common rescaling case both information is set and need to be simultaneously used. If the
+	 * input subtask subsumes the state of 3 old subtasks and a channel corresponds to 2 old channels, then there are
+	 * 6 virtual channels to be demultiplexed.
+	 */
+	public static class VirtualChannelMapping implements Serializable {
+		public static final PartitionMapping NO_CHANNEL_MAPPING = new PartitionMapping(emptyList());
+		public static final List<PartitionMapping> NO_PARTITIONS = emptyList();
+		public static final BitSet NO_SUBTASKS = new BitSet();
+		public static final VirtualChannelMapping NO_MAPPING = new VirtualChannelMapping(NO_SUBTASKS, NO_PARTITIONS);
+
+		/**
+		 * Set when several operator instances are merged into one.
+		 */
+		private final BitSet oldTaskInstances;
+
+		/**
+		 * Set when channels are merged because the connected operator has been rescaled.
+		 */
+		private final List<PartitionMapping> partitionMappings;
+
+		public VirtualChannelMapping(BitSet oldTaskInstances, List<PartitionMapping> partitionMappings) {
+			this.oldTaskInstances = oldTaskInstances;
+			this.partitionMappings = partitionMappings;
+		}
+
+		@Override
+		public boolean equals(Object o) {
+			if (this == o) {
+				return true;
+			}
+			if (o == null || getClass() != o.getClass()) {
+				return false;
+			}
+			final VirtualChannelMapping that = (VirtualChannelMapping) o;
+			return oldTaskInstances.equals(that.oldTaskInstances) &&
+				partitionMappings.equals(that.partitionMappings);
+		}
+
+		public int[] getOldTaskInstances(int defaultSubtask) {
+			return oldTaskInstances.equals(NO_SUBTASKS) ?
+				new int[] {defaultSubtask} :
+				oldTaskInstances.stream().toArray();
+		}
+
+		public PartitionMapping getPartitionMapping(int partitionIndex) {
+			if (partitionMappings.isEmpty()) {
+				return NO_CHANNEL_MAPPING;
+			}
+			return partitionMappings.get(partitionIndex);
+		}
+
+		@Override
+		public int hashCode() {
+			return Objects.hash(oldTaskInstances, partitionMappings);
+		}
+
+		@Override
+		public String toString() {
+			return "VirtualChannelMapping{" +
+				"oldTaskInstances=" + oldTaskInstances +
+				", partitionMappings=" + partitionMappings +
+				'}';
+		}
+	}
+
+	/**
+	 * Contains the fine-grain channel mappings that occur when a connected operator has been rescaled.
+	 */
+	public static class PartitionMapping implements Serializable {
+
+		/**
+		 * For each new channel (=index), all old channels are set.
+		 */
+		private final List<BitSet> newToOldChannelIndexes;

Review comment:
       If we have consecutive indexes from 0..n, what's wrong with `List<Set<Integer>>`? (I got the BitSet part)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517966691



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/CustomPartitionerWrapper.java
##########
@@ -55,6 +56,14 @@ public int selectChannel(SerializationDelegate<StreamRecord<T>> record) {
 		return partitioner.partition(key, numberOfChannels);
 	}
 
+	@Override
+	public ChannelStateRescaler getDownstreamChannelStateRescaler() {
+		// fully rely on filtering downstream
+		// note that custom partitioners are not officially supported - the user has to force rescaling
+		// in that case, we assume that the custom partitioner is deterministic
+		return ChannelStateRescaler.BROADCAST;

Review comment:
       This will not distribute state to the newer tasks in case of up-scaling, right?
   I think it can cause problems if the user-supplied partitioner relies on the new DOP.
   
   OTH, if not supported officially, we can leave it as it.
   
   WDYT?

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -64,81 +61,64 @@
 	/**
 	 * Snapshot from the {@link org.apache.flink.runtime.state.OperatorStateBackend}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<OperatorStateHandle> managedOperatorState;
 
 	/**
 	 * Snapshot written using {@link org.apache.flink.runtime.state.OperatorStateCheckpointOutputStream}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<OperatorStateHandle> rawOperatorState;
 
 	/**
 	 * Snapshot from {@link org.apache.flink.runtime.state.KeyedStateBackend}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<KeyedStateHandle> managedKeyedState;
 
 	/**
 	 * Snapshot written using {@link org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<KeyedStateHandle> rawKeyedState;
 
-	@Nonnull
 	private final StateObjectCollection<InputChannelStateHandle> inputChannelState;
 
-	@Nonnull
 	private final StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState;
 
 	/**
-	 * The state size. This is also part of the deserialized state handle.
-	 * We store it here in order to not deserialize the state handle when
-	 * gathering stats.
+	 * The subpartitions mappings per partition set when the output operator for a partition was rescaled. The key is
+	 * the partition id and the value contains all subtask indexes of the output operator before rescaling.
 	 */
-	private final long stateSize;
+	private final InflightDataRescalingDescriptor inputRescalingDescriptor;

Review comment:
       I think this field needs to be (de)serialized by `MetadataSerializer` as other fields are.
   
   (ditto `outputRescalingDescriptor`)

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ChannelRescalerRepartitioner.java
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.io.network.api.writer.ChannelStateRescaler;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.Iterables;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * A repartitioner that assigns the same channel state to multiple subtasks according to some mapping.
+ *
+ * <p>The replicated data will then be filtered before processing the record.
+ *
+ * <p>Note that channel mappings are cached for the same parallelism changes.
+ */
+public class ChannelRescalerRepartitioner<T> implements OperatorStateRepartitioner<T> {
+	private final ChannelStateRescaler channelStateRescaler;
+	private final Map<Tuple2<Integer, Integer>, Map<Integer, Set<Integer>>> newToOldMappingCache = new HashMap<>(2);

Review comment:
       nit:
   After removing the higher level cache, this field can also be computed eagerly think (as it will be used anyways if the is being object is constructed).

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/RescaledChannelsMapping.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import static java.util.Collections.emptyMap;
+
+/**
+ * Contains the fine-grain channel mappings that occur when a connected operator has been rescaled.
+ */
+public class RescaledChannelsMapping implements Serializable {
+	public static final RescaledChannelsMapping NO_CHANNEL_MAPPING = new RescaledChannelsMapping(emptyMap());
+
+	private static final long serialVersionUID = -8719670050630674631L;
+
+	/**
+	 * For each new channel (=index), all old channels are set.
+	 */
+	private final Map<Integer, Set<Integer>> newToOldChannelIndexes;
+
+	/**
+	 * For each old channel (=index), all new channels are set. Lazily calculated to keep {@link OperatorSubtaskState}
+	 * small in terms of serialization cost.
+	 */
+	private transient Map<Integer, Set<Integer>> oldToNewChannelIndexes;

Review comment:
       nit:
   This field makes this class non-thread-safe, which is not obvious for a pre-computed mapping.
   So I'd mark this class `@NotThreadSafe`.

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/ForwardPartitioner.java
##########
@@ -43,4 +44,9 @@ public int selectChannel(SerializationDelegate<StreamRecord<T>> record) {
 	public String toString() {
 		return "FORWARD";
 	}
+
+	@Override
+	public ChannelStateRescaler getDownstreamChannelStateRescaler() {
+		return ChannelStateRescaler.FIRST_CHANNEL;

Review comment:
       Is `FIRST_CHANNEL` here (i.e. during the assignment) the same as in `selectChannel` (i.e. during sending the data)?
   In other words, I think this will put all `InputChannel` state into a single subtask, while it should go to the same subtasks as before the rescaling.
   WDYT?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ee73393a13c999be626f7b8b98a170be88a093d9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077) 
   * c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise merged pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise merged pull request #13735:
URL: https://github.com/apache/flink/pull/13735


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517335535



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
##########
@@ -67,29 +83,33 @@
 	private final long restoreCheckpointId;
 	private final boolean allowNonRestoredState;
 
+	private final Map<IntermediateResult, TaskStateAssignment> consumerAssignment = new HashMap<>();
+	private final Map<ChannelStateRescaler, ChannelRescalerRepartitioner<Object>> rescalerRepartitioners =

Review comment:
       `rescalerRepartitioners` would save much more as `ChannelRescalerRepartitioner` internally caches as well.
   But I can take it out for now and we first measure before optimizing.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 1b1a34de33af36c301266d175cb72151b1bc51c4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179) 
   * 6bfcf5914a2742ce41f05b2b77b70f7df85fc696 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228",
       "triggerID" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8233",
       "triggerID" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8885",
       "triggerID" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfd138a5fda6a4eeb4544cc88f9e5660063ab084",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bfd138a5fda6a4eeb4544cc88f9e5660063ab084",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 9cbbcb24751268c0b5289b86606eb28baa1287eb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8885) 
   * bfd138a5fda6a4eeb4544cc88f9e5660063ab084 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517403672



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
##########
@@ -620,18 +758,112 @@ private static void checkStateMappingCompleteness(
 			chainOpParallelStates,
 			oldParallelism,
 			newParallelism);
+	}
+
+	static class TaskStateAssignment {

Review comment:
       1. Yes, but IIUC `TaskStateAssignment` could be added before
   2. Reviewing is one reason, but also in general why some helper code shouldn't reside in a separate file? Member lookup would be easier for example
   
   However, it's probably a matter of taste.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 1b1a34de33af36c301266d175cb72151b1bc51c4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517385040



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -327,4 +289,252 @@ public boolean hasState() {
 			|| inputChannelState.hasState()
 			|| resultSubpartitionState.hasState();
 	}
+
+	public static Builder builder() {
+		return new Builder();
+	}
+
+	/**
+	 * The builder for a new {@link OperatorSubtaskState} which can be obtained by {@link #builder()}.
+	 */
+	public static class Builder {
+		private StateObjectCollection<OperatorStateHandle> managedOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<OperatorStateHandle> rawOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> managedKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> rawKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<InputChannelStateHandle> inputChannelState = StateObjectCollection.empty();
+		private StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState = StateObjectCollection.empty();
+		private VirtualChannelMapping inputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+		private VirtualChannelMapping outputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+
+		private Builder() {
+		}
+
+		public Builder setManagedOperatorState(StateObjectCollection<OperatorStateHandle> managedOperatorState) {
+			this.managedOperatorState = checkNotNull(managedOperatorState);
+			return this;
+		}
+
+		public Builder setManagedOperatorState(OperatorStateHandle managedOperatorState) {
+			return setManagedOperatorState(StateObjectCollection.singleton(checkNotNull(managedOperatorState)));
+		}
+
+		public Builder setRawOperatorState(StateObjectCollection<OperatorStateHandle> rawOperatorState) {
+			this.rawOperatorState = checkNotNull(rawOperatorState);
+			return this;
+		}
+
+		public Builder setRawOperatorState(OperatorStateHandle rawOperatorState) {
+			return setRawOperatorState(StateObjectCollection.singleton(checkNotNull(rawOperatorState)));
+		}
+
+		public Builder setManagedKeyedState(StateObjectCollection<KeyedStateHandle> managedKeyedState) {
+			this.managedKeyedState = checkNotNull(managedKeyedState);
+			return this;
+		}
+
+		public Builder setManagedKeyedState(KeyedStateHandle managedKeyedState) {
+			return setManagedKeyedState(StateObjectCollection.singleton(checkNotNull(managedKeyedState)));
+		}
+
+		public Builder setRawKeyedState(StateObjectCollection<KeyedStateHandle> rawKeyedState) {
+			this.rawKeyedState = checkNotNull(rawKeyedState);
+			return this;
+		}
+
+		public Builder setRawKeyedState(KeyedStateHandle rawKeyedState) {
+			return setRawKeyedState(StateObjectCollection.singleton(checkNotNull(rawKeyedState)));
+		}
+
+		public Builder setInputChannelState(StateObjectCollection<InputChannelStateHandle> inputChannelState) {
+			this.inputChannelState = checkNotNull(inputChannelState);
+			return this;
+		}
+
+		public Builder setResultSubpartitionState(StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState) {
+			this.resultSubpartitionState = checkNotNull(resultSubpartitionState);
+			return this;
+		}
+
+		public Builder setInputChannelMappings(VirtualChannelMapping inputChannelMappings) {
+			this.inputChannelMappings = checkNotNull(inputChannelMappings);
+			return this;
+		}
+
+		public Builder setOutputChannelMappings(VirtualChannelMapping outputChannelMappings) {
+			this.outputChannelMappings = checkNotNull(outputChannelMappings);
+			return this;
+		}
+
+		public OperatorSubtaskState build() {
+			return new OperatorSubtaskState(
+				managedOperatorState,
+				rawOperatorState,
+				managedKeyedState,
+				rawKeyedState,
+				inputChannelState,
+				resultSubpartitionState,
+				inputChannelMappings,
+				outputChannelMappings);
+		}
+	}
+
+	/**
+	 * Captures ambiguous mappings of old channels to new channels.
+	 *
+	 * <p>For inputs, this mapping implies the following:
+	 * <li>
+	 *     <ul>{@link #oldTaskInstances} is set when there is a rescale on this task potentially leading to different
+	 *     key groups. Upstream task has a corresponding {@link #partitionMappings} where it sends data over
+	 *     virtual channel while specifying the channel index in the VirtualChannelSelector. This subtask then
+	 *     demultiplexes over the virtual subtask index.</ul>
+	 *     <ul>{@link #partitionMappings} is set when there is a downscale of the upstream task. Upstream task has
+	 *     a corresponding {@link #oldTaskInstances} where it sends data over virtual channel while specifying the
+	 *     subtask index in the VirtualChannelSelector. This subtask then demultiplexes over channel indexes.</ul>
+	 * </li>
+	 *
+	 * <p>For outputs, it's vice-versa. The information must be kept in sync but they are used in opposite ways for
+	 * multiplexing/demultiplexing.
+	 *
+	 * <p>Note that in the common rescaling case both information is set and need to be simultaneously used. If the
+	 * input subtask subsumes the state of 3 old subtasks and a channel corresponds to 2 old channels, then there are
+	 * 6 virtual channels to be demultiplexed.
+	 */
+	public static class VirtualChannelMapping implements Serializable {
+		public static final PartitionMapping NO_CHANNEL_MAPPING = new PartitionMapping(emptyList());
+		public static final List<PartitionMapping> NO_PARTITIONS = emptyList();
+		public static final BitSet NO_SUBTASKS = new BitSet();
+		public static final VirtualChannelMapping NO_MAPPING = new VirtualChannelMapping(NO_SUBTASKS, NO_PARTITIONS);
+
+		/**
+		 * Set when several operator instances are merged into one.
+		 */
+		private final BitSet oldTaskInstances;
+
+		/**
+		 * Set when channels are merged because the connected operator has been rescaled.
+		 */
+		private final List<PartitionMapping> partitionMappings;

Review comment:
       > Isn't array and array-list the same in complexity?
   
   Yes, but here it is `List`, which can be a `LinkedList`.
   
   If not arrays, then I'd go with Map, just because it makes obvious how elements are indexed and accessed. I don't think overhead will be visible here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228",
       "triggerID" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8233",
       "triggerID" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8885",
       "triggerID" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfd138a5fda6a4eeb4544cc88f9e5660063ab084",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9025",
       "triggerID" : "bfd138a5fda6a4eeb4544cc88f9e5660063ab084",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 9cbbcb24751268c0b5289b86606eb28baa1287eb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8885) 
   * bfd138a5fda6a4eeb4544cc88f9e5660063ab084 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9025) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 1b1a34de33af36c301266d175cb72151b1bc51c4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179) 
   * 6bfcf5914a2742ce41f05b2b77b70f7df85fc696 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518128784



##########
File path: flink-core/src/main/java/org/apache/flink/api/common/functions/Partitioner.java
##########
@@ -37,4 +40,20 @@
 	 * @return The partition index.
 	 */
 	int partition(K key, int numPartitions);
+
+	/**
+	 * Returns all partitions that need to be read to restore the given new partition. The partitioner is then
+	 * applied on the key of the restored record to filter all irrelevant records.
+	 *
+	 * <p>In particular, to create a partition X after rescaling, all partitions returned by this method are fully read
+	 * and the key of each record is then fed into {@link #partition(Object, int)} to check if it belongs to X.
+	 *
+	 * <p>The default implementation states that all partitions need to be scanned and should be overwritten to improve
+	 * performance.
+	 */
+	@PublicEvolving
+	default int[] rescaleIntersections(int newPartition, int oldNumPartitions, int newNumPartitions) {
+		// any old partition may contain a record that should be in the new partition after rescaling
+		return IntStream.range(0, oldNumPartitions).toArray();
+	}

Review comment:
       Removed from `CustomerPartitioner`. We officially do not support it (and inofficially just use full replication and downstream filtering).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228",
       "triggerID" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8233",
       "triggerID" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8885",
       "triggerID" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfd138a5fda6a4eeb4544cc88f9e5660063ab084",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9025",
       "triggerID" : "bfd138a5fda6a4eeb4544cc88f9e5660063ab084",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * bfd138a5fda6a4eeb4544cc88f9e5660063ab084 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9025) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228",
       "triggerID" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8233",
       "triggerID" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 25b0c065fae8c72fa607c5b6f47376ff3e6f8568 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8233) 
   * 9cbbcb24751268c0b5289b86606eb28baa1287eb UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228",
       "triggerID" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224) 
   * 88a90896e0cf1c7bf27fe1aa40158c7280c5fa58 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517415880



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java
##########
@@ -692,6 +692,8 @@ private void connect(Integer headOfChain, StreamEdge edge) {
 		}
 		// set strategy name so that web interface can show it.
 		jobEdge.setShipStrategyName(partitioner.toString());
+		jobEdge.setDownstreamChannelStateRescaler(partitioner.getUpstreamChannelStateRescaler());
+		jobEdge.setUpstreamChannelStateRescaler(partitioner.getDownstreamChannelStateRescaler());

Review comment:
       No worries, thanks for updating.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224) 
   * 88a90896e0cf1c7bf27fe1aa40158c7280c5fa58 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517332664



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/ForwardPartitioner.java
##########
@@ -43,4 +44,14 @@ public int selectChannel(SerializationDelegate<StreamRecord<T>> record) {
 	public String toString() {
 		return "FORWARD";
 	}
+
+	@Override
+	public ChannelStateRescaler getUpstreamChannelStateRescaler() {
+		return ChannelStateRescaler.FIRST_CHANNEL;
+	}
+
+	@Override
+	public ChannelStateRescaler getDownstreamChannelStateRescaler() {
+		return ChannelStateRescaler.ROUND_ROBIN;

Review comment:
       You are right that things should match. However, for channel state there are two dimensions on how to scale: on subtask-level and on channel-level. If you think of a keyed exchange, a rescale on downstream means that the key ranges are changing. A rescale on upstream just means that we need to redistribute the channel state to an arbitrary other subtask.
   That's true for all exchanges except broadcast (where we need to discard extra state). I could also opt to just have a `boolean shouldRedistributeUpstreamState` instead of using `ChannelStateRescaler`. WDYT?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r516943950



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -63,31 +61,25 @@
 	/**
 	 * Snapshot from the {@link org.apache.flink.runtime.state.OperatorStateBackend}.
 	 */
-	@Nonnull

Review comment:
       I like the change. Especially given that there are already `checkNotNull` in the constructor.
   (we don't have to remove again it if someone re-adds it)
   
   P.S.: is `checkNotNull` missing for the new fields?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e081fecba324a0e35017f4b89cc0499c9112a768 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 6bfcf5914a2742ce41f05b2b77b70f7df85fc696 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198) 
   * 470d466cfe4455aefd5a302cedd92b57750f2c57 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 5e5340241b5811243463f96dda7455b2869cf268 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223) 
   * cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517414170



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/ForwardPartitioner.java
##########
@@ -43,4 +44,14 @@ public int selectChannel(SerializationDelegate<StreamRecord<T>> record) {
 	public String toString() {
 		return "FORWARD";
 	}
+
+	@Override
+	public ChannelStateRescaler getUpstreamChannelStateRescaler() {
+		return ChannelStateRescaler.FIRST_CHANNEL;
+	}
+
+	@Override
+	public ChannelStateRescaler getDownstreamChannelStateRescaler() {
+		return ChannelStateRescaler.ROUND_ROBIN;

Review comment:
       I agree about using `ROUND_ROBIN` for the upstream. 
   My concern was mostly about the dowstream (which should be resolved by addressing the comment [below](https://github.com/apache/flink/pull/13735#discussion_r517176106)).
   
   As for `boolean`, I'd prefer `ChannelStateRescaler` as it's more flexible and consistent (with downstream).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 6bfcf5914a2742ce41f05b2b77b70f7df85fc696 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517152644



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -327,4 +289,252 @@ public boolean hasState() {
 			|| inputChannelState.hasState()
 			|| resultSubpartitionState.hasState();
 	}
+
+	public static Builder builder() {
+		return new Builder();
+	}
+
+	/**
+	 * The builder for a new {@link OperatorSubtaskState} which can be obtained by {@link #builder()}.
+	 */
+	public static class Builder {
+		private StateObjectCollection<OperatorStateHandle> managedOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<OperatorStateHandle> rawOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> managedKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> rawKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<InputChannelStateHandle> inputChannelState = StateObjectCollection.empty();
+		private StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState = StateObjectCollection.empty();
+		private VirtualChannelMapping inputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+		private VirtualChannelMapping outputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+
+		private Builder() {
+		}
+
+		public Builder setManagedOperatorState(StateObjectCollection<OperatorStateHandle> managedOperatorState) {
+			this.managedOperatorState = checkNotNull(managedOperatorState);
+			return this;
+		}
+
+		public Builder setManagedOperatorState(OperatorStateHandle managedOperatorState) {
+			return setManagedOperatorState(StateObjectCollection.singleton(checkNotNull(managedOperatorState)));
+		}
+
+		public Builder setRawOperatorState(StateObjectCollection<OperatorStateHandle> rawOperatorState) {
+			this.rawOperatorState = checkNotNull(rawOperatorState);
+			return this;
+		}
+
+		public Builder setRawOperatorState(OperatorStateHandle rawOperatorState) {
+			return setRawOperatorState(StateObjectCollection.singleton(checkNotNull(rawOperatorState)));
+		}
+
+		public Builder setManagedKeyedState(StateObjectCollection<KeyedStateHandle> managedKeyedState) {
+			this.managedKeyedState = checkNotNull(managedKeyedState);
+			return this;
+		}
+
+		public Builder setManagedKeyedState(KeyedStateHandle managedKeyedState) {
+			return setManagedKeyedState(StateObjectCollection.singleton(checkNotNull(managedKeyedState)));
+		}
+
+		public Builder setRawKeyedState(StateObjectCollection<KeyedStateHandle> rawKeyedState) {
+			this.rawKeyedState = checkNotNull(rawKeyedState);
+			return this;
+		}
+
+		public Builder setRawKeyedState(KeyedStateHandle rawKeyedState) {
+			return setRawKeyedState(StateObjectCollection.singleton(checkNotNull(rawKeyedState)));
+		}
+
+		public Builder setInputChannelState(StateObjectCollection<InputChannelStateHandle> inputChannelState) {
+			this.inputChannelState = checkNotNull(inputChannelState);
+			return this;
+		}
+
+		public Builder setResultSubpartitionState(StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState) {
+			this.resultSubpartitionState = checkNotNull(resultSubpartitionState);
+			return this;
+		}
+
+		public Builder setInputChannelMappings(VirtualChannelMapping inputChannelMappings) {
+			this.inputChannelMappings = checkNotNull(inputChannelMappings);
+			return this;
+		}
+
+		public Builder setOutputChannelMappings(VirtualChannelMapping outputChannelMappings) {
+			this.outputChannelMappings = checkNotNull(outputChannelMappings);
+			return this;
+		}
+
+		public OperatorSubtaskState build() {
+			return new OperatorSubtaskState(
+				managedOperatorState,
+				rawOperatorState,
+				managedKeyedState,
+				rawKeyedState,
+				inputChannelState,
+				resultSubpartitionState,
+				inputChannelMappings,
+				outputChannelMappings);
+		}
+	}
+
+	/**
+	 * Captures ambiguous mappings of old channels to new channels.
+	 *
+	 * <p>For inputs, this mapping implies the following:
+	 * <li>
+	 *     <ul>{@link #oldTaskInstances} is set when there is a rescale on this task potentially leading to different
+	 *     key groups. Upstream task has a corresponding {@link #partitionMappings} where it sends data over
+	 *     virtual channel while specifying the channel index in the VirtualChannelSelector. This subtask then
+	 *     demultiplexes over the virtual subtask index.</ul>
+	 *     <ul>{@link #partitionMappings} is set when there is a downscale of the upstream task. Upstream task has
+	 *     a corresponding {@link #oldTaskInstances} where it sends data over virtual channel while specifying the
+	 *     subtask index in the VirtualChannelSelector. This subtask then demultiplexes over channel indexes.</ul>
+	 * </li>
+	 *
+	 * <p>For outputs, it's vice-versa. The information must be kept in sync but they are used in opposite ways for
+	 * multiplexing/demultiplexing.
+	 *
+	 * <p>Note that in the common rescaling case both information is set and need to be simultaneously used. If the
+	 * input subtask subsumes the state of 3 old subtasks and a channel corresponds to 2 old channels, then there are
+	 * 6 virtual channels to be demultiplexed.
+	 */
+	public static class VirtualChannelMapping implements Serializable {

Review comment:
       hm, it's all about channel state as part of the operator state. How about `ChannelStateMapping`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518416601



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
##########
@@ -620,18 +758,112 @@ private static void checkStateMappingCompleteness(
 			chainOpParallelStates,
 			oldParallelism,
 			newParallelism);
+	}
+
+	static class TaskStateAssignment {

Review comment:
       Thanks a lot! To me, it's easier to understand changes now.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518178453



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -64,81 +61,64 @@
 	/**
 	 * Snapshot from the {@link org.apache.flink.runtime.state.OperatorStateBackend}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<OperatorStateHandle> managedOperatorState;
 
 	/**
 	 * Snapshot written using {@link org.apache.flink.runtime.state.OperatorStateCheckpointOutputStream}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<OperatorStateHandle> rawOperatorState;
 
 	/**
 	 * Snapshot from {@link org.apache.flink.runtime.state.KeyedStateBackend}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<KeyedStateHandle> managedKeyedState;
 
 	/**
 	 * Snapshot written using {@link org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<KeyedStateHandle> rawKeyedState;
 
-	@Nonnull
 	private final StateObjectCollection<InputChannelStateHandle> inputChannelState;
 
-	@Nonnull
 	private final StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState;
 
 	/**
-	 * The state size. This is also part of the deserialized state handle.
-	 * We store it here in order to not deserialize the state handle when
-	 * gathering stats.
+	 * The subpartitions mappings per partition set when the output operator for a partition was rescaled. The key is
+	 * the partition id and the value contains all subtask indexes of the output operator before rescaling.
 	 */
-	private final long stateSize;
+	private final InflightDataRescalingDescriptor inputRescalingDescriptor;

Review comment:
       Hm in the serialized form it would always be empty, so I'm not sure what the advantage is (it can only be filled when you know the post-scale parallelism, which you wouldn't know in a checkpoint).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517153849



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -327,4 +289,252 @@ public boolean hasState() {
 			|| inputChannelState.hasState()
 			|| resultSubpartitionState.hasState();
 	}
+
+	public static Builder builder() {
+		return new Builder();
+	}
+
+	/**
+	 * The builder for a new {@link OperatorSubtaskState} which can be obtained by {@link #builder()}.
+	 */
+	public static class Builder {
+		private StateObjectCollection<OperatorStateHandle> managedOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<OperatorStateHandle> rawOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> managedKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> rawKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<InputChannelStateHandle> inputChannelState = StateObjectCollection.empty();
+		private StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState = StateObjectCollection.empty();
+		private VirtualChannelMapping inputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+		private VirtualChannelMapping outputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+
+		private Builder() {
+		}
+
+		public Builder setManagedOperatorState(StateObjectCollection<OperatorStateHandle> managedOperatorState) {
+			this.managedOperatorState = checkNotNull(managedOperatorState);
+			return this;
+		}
+
+		public Builder setManagedOperatorState(OperatorStateHandle managedOperatorState) {
+			return setManagedOperatorState(StateObjectCollection.singleton(checkNotNull(managedOperatorState)));
+		}
+
+		public Builder setRawOperatorState(StateObjectCollection<OperatorStateHandle> rawOperatorState) {
+			this.rawOperatorState = checkNotNull(rawOperatorState);
+			return this;
+		}
+
+		public Builder setRawOperatorState(OperatorStateHandle rawOperatorState) {
+			return setRawOperatorState(StateObjectCollection.singleton(checkNotNull(rawOperatorState)));
+		}
+
+		public Builder setManagedKeyedState(StateObjectCollection<KeyedStateHandle> managedKeyedState) {
+			this.managedKeyedState = checkNotNull(managedKeyedState);
+			return this;
+		}
+
+		public Builder setManagedKeyedState(KeyedStateHandle managedKeyedState) {
+			return setManagedKeyedState(StateObjectCollection.singleton(checkNotNull(managedKeyedState)));
+		}
+
+		public Builder setRawKeyedState(StateObjectCollection<KeyedStateHandle> rawKeyedState) {
+			this.rawKeyedState = checkNotNull(rawKeyedState);
+			return this;
+		}
+
+		public Builder setRawKeyedState(KeyedStateHandle rawKeyedState) {
+			return setRawKeyedState(StateObjectCollection.singleton(checkNotNull(rawKeyedState)));
+		}
+
+		public Builder setInputChannelState(StateObjectCollection<InputChannelStateHandle> inputChannelState) {
+			this.inputChannelState = checkNotNull(inputChannelState);
+			return this;
+		}
+
+		public Builder setResultSubpartitionState(StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState) {
+			this.resultSubpartitionState = checkNotNull(resultSubpartitionState);
+			return this;
+		}
+
+		public Builder setInputChannelMappings(VirtualChannelMapping inputChannelMappings) {
+			this.inputChannelMappings = checkNotNull(inputChannelMappings);
+			return this;
+		}
+
+		public Builder setOutputChannelMappings(VirtualChannelMapping outputChannelMappings) {
+			this.outputChannelMappings = checkNotNull(outputChannelMappings);
+			return this;
+		}
+
+		public OperatorSubtaskState build() {
+			return new OperatorSubtaskState(
+				managedOperatorState,
+				rawOperatorState,
+				managedKeyedState,
+				rawKeyedState,
+				inputChannelState,
+				resultSubpartitionState,
+				inputChannelMappings,
+				outputChannelMappings);
+		}
+	}
+
+	/**
+	 * Captures ambiguous mappings of old channels to new channels.
+	 *
+	 * <p>For inputs, this mapping implies the following:
+	 * <li>
+	 *     <ul>{@link #oldTaskInstances} is set when there is a rescale on this task potentially leading to different
+	 *     key groups. Upstream task has a corresponding {@link #partitionMappings} where it sends data over
+	 *     virtual channel while specifying the channel index in the VirtualChannelSelector. This subtask then
+	 *     demultiplexes over the virtual subtask index.</ul>
+	 *     <ul>{@link #partitionMappings} is set when there is a downscale of the upstream task. Upstream task has
+	 *     a corresponding {@link #oldTaskInstances} where it sends data over virtual channel while specifying the
+	 *     subtask index in the VirtualChannelSelector. This subtask then demultiplexes over channel indexes.</ul>
+	 * </li>
+	 *
+	 * <p>For outputs, it's vice-versa. The information must be kept in sync but they are used in opposite ways for
+	 * multiplexing/demultiplexing.
+	 *
+	 * <p>Note that in the common rescaling case both information is set and need to be simultaneously used. If the
+	 * input subtask subsumes the state of 3 old subtasks and a channel corresponds to 2 old channels, then there are
+	 * 6 virtual channels to be demultiplexed.
+	 */
+	public static class VirtualChannelMapping implements Serializable {
+		public static final PartitionMapping NO_CHANNEL_MAPPING = new PartitionMapping(emptyList());
+		public static final List<PartitionMapping> NO_PARTITIONS = emptyList();
+		public static final BitSet NO_SUBTASKS = new BitSet();
+		public static final VirtualChannelMapping NO_MAPPING = new VirtualChannelMapping(NO_SUBTASKS, NO_PARTITIONS);
+
+		/**
+		 * Set when several operator instances are merged into one.
+		 */
+		private final BitSet oldTaskInstances;

Review comment:
       I used it for memory optimization but it might have been prematurely:
   
   `Set<Integer>` uses 16 bytes per entry + 4 bytes for boxing + hash table overhead
   `BitSet` would be more efficient at least for all entries up to 24 bytes (so 24 channels).
   But I see that it's not scaling well, especially since `Set<Integer>` should be mostly 1 or 2 entries (so ~50 bytes).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228",
       "triggerID" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8233",
       "triggerID" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8885",
       "triggerID" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfd138a5fda6a4eeb4544cc88f9e5660063ab084",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9025",
       "triggerID" : "bfd138a5fda6a4eeb4544cc88f9e5660063ab084",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "722547278",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c95db355becc1079be6287aa7d165e7590f9fcf6",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9104",
       "triggerID" : "722547278",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c95db355becc1079be6287aa7d165e7590f9fcf6",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9104",
       "triggerID" : "c95db355becc1079be6287aa7d165e7590f9fcf6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   *  Unknown: [CANCELED](TBD) 
   * c95db355becc1079be6287aa7d165e7590f9fcf6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9104) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228",
       "triggerID" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8233",
       "triggerID" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8885",
       "triggerID" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfd138a5fda6a4eeb4544cc88f9e5660063ab084",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9025",
       "triggerID" : "bfd138a5fda6a4eeb4544cc88f9e5660063ab084",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c95db355becc1079be6287aa7d165e7590f9fcf6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "722547278",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "722547278",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c95db355becc1079be6287aa7d165e7590f9fcf6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c95db355becc1079be6287aa7d165e7590f9fcf6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   *  Unknown: [CANCELED](TBD) 
   * c95db355becc1079be6287aa7d165e7590f9fcf6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517372334



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
##########
@@ -620,18 +758,112 @@ private static void checkStateMappingCompleteness(
 			chainOpParallelStates,
 			oldParallelism,
 			newParallelism);
+	}
+
+	static class TaskStateAssignment {

Review comment:
       1. Isn't it introduced by the two-pass refactoring commit?
   2. It's a purely internal class of `StateAssignmentOperation`. Would it be easier because of side-by-side view or is there a different reason?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-722547278


   @flinkbot run azure


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518128144



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/ChannelRescaler.java
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.api.writer;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Serializable;
+import java.util.BitSet;
+
+/**
+ * The {@link ChannelRescaler} narrows down the channels that need to be read during rescaling to recover from a
+ * particular channel when in-flight data has been stored in the checkpoint.
+ */
+@Internal
+public interface ChannelRescaler extends Serializable {
+	/**
+	 * Returns all old channel indexes that need to be read to restore all buffers for the given new channel index on
+	 * rescale.
+	 */
+	BitSet rescaleIntersections(int newChannelIndex, int oldNumberOfChannels, int newNumberOfChannels);

Review comment:
       Removed from `Partitioner`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517663178



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -327,4 +289,252 @@ public boolean hasState() {
 			|| inputChannelState.hasState()
 			|| resultSubpartitionState.hasState();
 	}
+
+	public static Builder builder() {
+		return new Builder();
+	}
+
+	/**
+	 * The builder for a new {@link OperatorSubtaskState} which can be obtained by {@link #builder()}.
+	 */
+	public static class Builder {
+		private StateObjectCollection<OperatorStateHandle> managedOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<OperatorStateHandle> rawOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> managedKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> rawKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<InputChannelStateHandle> inputChannelState = StateObjectCollection.empty();
+		private StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState = StateObjectCollection.empty();
+		private VirtualChannelMapping inputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+		private VirtualChannelMapping outputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+
+		private Builder() {
+		}
+
+		public Builder setManagedOperatorState(StateObjectCollection<OperatorStateHandle> managedOperatorState) {
+			this.managedOperatorState = checkNotNull(managedOperatorState);
+			return this;
+		}
+
+		public Builder setManagedOperatorState(OperatorStateHandle managedOperatorState) {
+			return setManagedOperatorState(StateObjectCollection.singleton(checkNotNull(managedOperatorState)));
+		}
+
+		public Builder setRawOperatorState(StateObjectCollection<OperatorStateHandle> rawOperatorState) {
+			this.rawOperatorState = checkNotNull(rawOperatorState);
+			return this;
+		}
+
+		public Builder setRawOperatorState(OperatorStateHandle rawOperatorState) {
+			return setRawOperatorState(StateObjectCollection.singleton(checkNotNull(rawOperatorState)));
+		}
+
+		public Builder setManagedKeyedState(StateObjectCollection<KeyedStateHandle> managedKeyedState) {
+			this.managedKeyedState = checkNotNull(managedKeyedState);
+			return this;
+		}
+
+		public Builder setManagedKeyedState(KeyedStateHandle managedKeyedState) {
+			return setManagedKeyedState(StateObjectCollection.singleton(checkNotNull(managedKeyedState)));
+		}
+
+		public Builder setRawKeyedState(StateObjectCollection<KeyedStateHandle> rawKeyedState) {
+			this.rawKeyedState = checkNotNull(rawKeyedState);
+			return this;
+		}
+
+		public Builder setRawKeyedState(KeyedStateHandle rawKeyedState) {
+			return setRawKeyedState(StateObjectCollection.singleton(checkNotNull(rawKeyedState)));
+		}
+
+		public Builder setInputChannelState(StateObjectCollection<InputChannelStateHandle> inputChannelState) {
+			this.inputChannelState = checkNotNull(inputChannelState);
+			return this;
+		}
+
+		public Builder setResultSubpartitionState(StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState) {
+			this.resultSubpartitionState = checkNotNull(resultSubpartitionState);
+			return this;
+		}
+
+		public Builder setInputChannelMappings(VirtualChannelMapping inputChannelMappings) {
+			this.inputChannelMappings = checkNotNull(inputChannelMappings);
+			return this;
+		}
+
+		public Builder setOutputChannelMappings(VirtualChannelMapping outputChannelMappings) {
+			this.outputChannelMappings = checkNotNull(outputChannelMappings);
+			return this;
+		}
+
+		public OperatorSubtaskState build() {
+			return new OperatorSubtaskState(
+				managedOperatorState,
+				rawOperatorState,
+				managedKeyedState,
+				rawKeyedState,
+				inputChannelState,
+				resultSubpartitionState,
+				inputChannelMappings,
+				outputChannelMappings);
+		}
+	}
+
+	/**
+	 * Captures ambiguous mappings of old channels to new channels.
+	 *
+	 * <p>For inputs, this mapping implies the following:
+	 * <li>
+	 *     <ul>{@link #oldTaskInstances} is set when there is a rescale on this task potentially leading to different
+	 *     key groups. Upstream task has a corresponding {@link #partitionMappings} where it sends data over
+	 *     virtual channel while specifying the channel index in the VirtualChannelSelector. This subtask then
+	 *     demultiplexes over the virtual subtask index.</ul>
+	 *     <ul>{@link #partitionMappings} is set when there is a downscale of the upstream task. Upstream task has
+	 *     a corresponding {@link #oldTaskInstances} where it sends data over virtual channel while specifying the
+	 *     subtask index in the VirtualChannelSelector. This subtask then demultiplexes over channel indexes.</ul>
+	 * </li>
+	 *
+	 * <p>For outputs, it's vice-versa. The information must be kept in sync but they are used in opposite ways for
+	 * multiplexing/demultiplexing.
+	 *
+	 * <p>Note that in the common rescaling case both information is set and need to be simultaneously used. If the
+	 * input subtask subsumes the state of 3 old subtasks and a channel corresponds to 2 old channels, then there are
+	 * 6 virtual channels to be demultiplexed.
+	 */
+	public static class VirtualChannelMapping implements Serializable {

Review comment:
       I went with `InflightDataRescalingDescriptor` for now.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517393887



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -327,4 +289,252 @@ public boolean hasState() {
 			|| inputChannelState.hasState()
 			|| resultSubpartitionState.hasState();
 	}
+
+	public static Builder builder() {
+		return new Builder();
+	}
+
+	/**
+	 * The builder for a new {@link OperatorSubtaskState} which can be obtained by {@link #builder()}.
+	 */
+	public static class Builder {
+		private StateObjectCollection<OperatorStateHandle> managedOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<OperatorStateHandle> rawOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> managedKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> rawKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<InputChannelStateHandle> inputChannelState = StateObjectCollection.empty();
+		private StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState = StateObjectCollection.empty();
+		private VirtualChannelMapping inputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+		private VirtualChannelMapping outputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+
+		private Builder() {
+		}
+
+		public Builder setManagedOperatorState(StateObjectCollection<OperatorStateHandle> managedOperatorState) {
+			this.managedOperatorState = checkNotNull(managedOperatorState);
+			return this;
+		}
+
+		public Builder setManagedOperatorState(OperatorStateHandle managedOperatorState) {
+			return setManagedOperatorState(StateObjectCollection.singleton(checkNotNull(managedOperatorState)));
+		}
+
+		public Builder setRawOperatorState(StateObjectCollection<OperatorStateHandle> rawOperatorState) {
+			this.rawOperatorState = checkNotNull(rawOperatorState);
+			return this;
+		}
+
+		public Builder setRawOperatorState(OperatorStateHandle rawOperatorState) {
+			return setRawOperatorState(StateObjectCollection.singleton(checkNotNull(rawOperatorState)));
+		}
+
+		public Builder setManagedKeyedState(StateObjectCollection<KeyedStateHandle> managedKeyedState) {
+			this.managedKeyedState = checkNotNull(managedKeyedState);
+			return this;
+		}
+
+		public Builder setManagedKeyedState(KeyedStateHandle managedKeyedState) {
+			return setManagedKeyedState(StateObjectCollection.singleton(checkNotNull(managedKeyedState)));
+		}
+
+		public Builder setRawKeyedState(StateObjectCollection<KeyedStateHandle> rawKeyedState) {
+			this.rawKeyedState = checkNotNull(rawKeyedState);
+			return this;
+		}
+
+		public Builder setRawKeyedState(KeyedStateHandle rawKeyedState) {
+			return setRawKeyedState(StateObjectCollection.singleton(checkNotNull(rawKeyedState)));
+		}
+
+		public Builder setInputChannelState(StateObjectCollection<InputChannelStateHandle> inputChannelState) {
+			this.inputChannelState = checkNotNull(inputChannelState);
+			return this;
+		}
+
+		public Builder setResultSubpartitionState(StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState) {
+			this.resultSubpartitionState = checkNotNull(resultSubpartitionState);
+			return this;
+		}
+
+		public Builder setInputChannelMappings(VirtualChannelMapping inputChannelMappings) {
+			this.inputChannelMappings = checkNotNull(inputChannelMappings);
+			return this;
+		}
+
+		public Builder setOutputChannelMappings(VirtualChannelMapping outputChannelMappings) {
+			this.outputChannelMappings = checkNotNull(outputChannelMappings);
+			return this;
+		}
+
+		public OperatorSubtaskState build() {
+			return new OperatorSubtaskState(
+				managedOperatorState,
+				rawOperatorState,
+				managedKeyedState,
+				rawKeyedState,
+				inputChannelState,
+				resultSubpartitionState,
+				inputChannelMappings,
+				outputChannelMappings);
+		}
+	}
+
+	/**
+	 * Captures ambiguous mappings of old channels to new channels.
+	 *
+	 * <p>For inputs, this mapping implies the following:
+	 * <li>
+	 *     <ul>{@link #oldTaskInstances} is set when there is a rescale on this task potentially leading to different
+	 *     key groups. Upstream task has a corresponding {@link #partitionMappings} where it sends data over
+	 *     virtual channel while specifying the channel index in the VirtualChannelSelector. This subtask then
+	 *     demultiplexes over the virtual subtask index.</ul>
+	 *     <ul>{@link #partitionMappings} is set when there is a downscale of the upstream task. Upstream task has
+	 *     a corresponding {@link #oldTaskInstances} where it sends data over virtual channel while specifying the
+	 *     subtask index in the VirtualChannelSelector. This subtask then demultiplexes over channel indexes.</ul>
+	 * </li>
+	 *
+	 * <p>For outputs, it's vice-versa. The information must be kept in sync but they are used in opposite ways for
+	 * multiplexing/demultiplexing.
+	 *
+	 * <p>Note that in the common rescaling case both information is set and need to be simultaneously used. If the
+	 * input subtask subsumes the state of 3 old subtasks and a channel corresponds to 2 old channels, then there are
+	 * 6 virtual channels to be demultiplexed.
+	 */
+	public static class VirtualChannelMapping implements Serializable {
+		public static final PartitionMapping NO_CHANNEL_MAPPING = new PartitionMapping(emptyList());
+		public static final List<PartitionMapping> NO_PARTITIONS = emptyList();
+		public static final BitSet NO_SUBTASKS = new BitSet();
+		public static final VirtualChannelMapping NO_MAPPING = new VirtualChannelMapping(NO_SUBTASKS, NO_PARTITIONS);
+
+		/**
+		 * Set when several operator instances are merged into one.
+		 */
+		private final BitSet oldTaskInstances;
+
+		/**
+		 * Set when channels are merged because the connected operator has been rescaled.
+		 */
+		private final List<PartitionMapping> partitionMappings;
+
+		public VirtualChannelMapping(BitSet oldTaskInstances, List<PartitionMapping> partitionMappings) {
+			this.oldTaskInstances = oldTaskInstances;
+			this.partitionMappings = partitionMappings;
+		}
+
+		@Override
+		public boolean equals(Object o) {
+			if (this == o) {
+				return true;
+			}
+			if (o == null || getClass() != o.getClass()) {
+				return false;
+			}
+			final VirtualChannelMapping that = (VirtualChannelMapping) o;
+			return oldTaskInstances.equals(that.oldTaskInstances) &&
+				partitionMappings.equals(that.partitionMappings);
+		}
+
+		public int[] getOldTaskInstances(int defaultSubtask) {
+			return oldTaskInstances.equals(NO_SUBTASKS) ?
+				new int[] {defaultSubtask} :
+				oldTaskInstances.stream().toArray();
+		}
+
+		public PartitionMapping getPartitionMapping(int partitionIndex) {
+			if (partitionMappings.isEmpty()) {
+				return NO_CHANNEL_MAPPING;
+			}
+			return partitionMappings.get(partitionIndex);
+		}
+
+		@Override
+		public int hashCode() {
+			return Objects.hash(oldTaskInstances, partitionMappings);
+		}
+
+		@Override
+		public String toString() {
+			return "VirtualChannelMapping{" +
+				"oldTaskInstances=" + oldTaskInstances +
+				", partitionMappings=" + partitionMappings +
+				'}';
+		}
+	}
+
+	/**
+	 * Contains the fine-grain channel mappings that occur when a connected operator has been rescaled.
+	 */
+	public static class PartitionMapping implements Serializable {
+
+		/**
+		 * For each new channel (=index), all old channels are set.
+		 */
+		private final List<BitSet> newToOldChannelIndexes;

Review comment:
       Nothing wrong performance-wise, but `Map<Int, ...>` just better captures the purpose:
   ```
   newToOldChannelIndexes.get(newChannelIndex)
   ```
   
   For the list I'd assume some iteration.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518423944



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ChannelRescalerRepartitioner.java
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.io.network.api.writer.ChannelStateRescaler;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.Iterables;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * A repartitioner that assigns the same channel state to multiple subtasks according to some mapping.
+ *
+ * <p>The replicated data will then be filtered before processing the record.
+ *
+ * <p>Note that channel mappings are cached for the same parallelism changes.
+ */
+public class ChannelRescalerRepartitioner<T> implements OperatorStateRepartitioner<T> {
+	private final ChannelStateRescaler channelStateRescaler;
+	private final Map<Tuple2<Integer, Integer>, Map<Integer, Set<Integer>>> newToOldMappingCache = new HashMap<>(2);
+
+	public ChannelRescalerRepartitioner(ChannelStateRescaler channelStateRescaler) {
+		this.channelStateRescaler = channelStateRescaler;
+	}
+
+	private static <T> List<T> getOldState(List<List<T>> previousParallelSubtaskStates, Set<Integer> oldIndexes) {
+		switch (oldIndexes.size()) {
+			case 0:
+				return Collections.emptyList();
+			case 1:
+				return previousParallelSubtaskStates.get(Iterables.getOnlyElement(oldIndexes));
+			default:
+				return oldIndexes.stream()
+					.flatMap(oldIndex -> previousParallelSubtaskStates.get(oldIndex).stream())
+					.collect(Collectors.toList());
+		}
+	}
+
+	protected Map<Integer, Set<Integer>> createNewToOldMapping(int oldParallelism, int newParallelism) {
+		return IntStream.range(0, newParallelism).boxed().
+			collect(Collectors.toMap(
+				Function.identity(),
+				channelIndex -> channelStateRescaler.getOldChannels(
+					channelIndex,
+					oldParallelism,
+					newParallelism)));
+	}
+
+	@Override
+	public List<List<T>> repartitionState(
+			List<List<T>> previousParallelSubtaskStates,
+			int oldParallelism,
+			int newParallelism) {
+		final Map<Integer, Set<Integer>> newToOldMapping = getNewToOldMapping(oldParallelism, newParallelism);
+		return IntStream.range(0, newParallelism)
+			.mapToObj(newIndex -> getOldState(previousParallelSubtaskStates, newToOldMapping.get(newIndex)))
+			.collect(Collectors.toList());

Review comment:
       Even better!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228",
       "triggerID" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8233",
       "triggerID" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224) 
   * 88a90896e0cf1c7bf27fe1aa40158c7280c5fa58 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228) 
   * 25b0c065fae8c72fa607c5b6f47376ff3e6f8568 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8233) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116) 
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 7c42ceb0f3c54b0ab63f053ce91a20c4866119d1 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517406747



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/AbstractChannelStateHandle.java
##########
@@ -46,6 +46,11 @@
 	private final List<Long> offsets;
 	private final long size;
 
+	/**
+	 * The original subtask index before rescaling recovery.
+	 */
+	private int originalSubtaskIndex;

Review comment:
       That means that after recovery it's not valid until explicitly set, right?
   I think it's quite error-prone. I'd at least document this behavior and mark it deprecated to give a warning.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518417599



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/CustomPartitionerWrapper.java
##########
@@ -55,6 +56,14 @@ public int selectChannel(SerializationDelegate<StreamRecord<T>> record) {
 		return partitioner.partition(key, numberOfChannels);
 	}
 
+	@Override
+	public ChannelStateRescaler getDownstreamChannelStateRescaler() {
+		// fully rely on filtering downstream
+		// note that custom partitioners are not officially supported - the user has to force rescaling
+		// in that case, we assume that the custom partitioner is deterministic
+		return ChannelStateRescaler.BROADCAST;

Review comment:
       Yes, you are right.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228",
       "triggerID" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8233",
       "triggerID" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 25b0c065fae8c72fa607c5b6f47376ff3e6f8568 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8233) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518061766



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ChannelRescalerRepartitioner.java
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.io.network.api.writer.ChannelStateRescaler;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.Iterables;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * A repartitioner that assigns the same channel state to multiple subtasks according to some mapping.
+ *
+ * <p>The replicated data will then be filtered before processing the record.
+ *
+ * <p>Note that channel mappings are cached for the same parallelism changes.
+ */
+public class ChannelRescalerRepartitioner<T> implements OperatorStateRepartitioner<T> {
+	private final ChannelStateRescaler channelStateRescaler;
+	private final Map<Tuple2<Integer, Integer>, Map<Integer, Set<Integer>>> newToOldMappingCache = new HashMap<>(2);
+
+	public ChannelRescalerRepartitioner(ChannelStateRescaler channelStateRescaler) {
+		this.channelStateRescaler = channelStateRescaler;
+	}
+
+	private static <T> List<T> getOldState(List<List<T>> previousParallelSubtaskStates, Set<Integer> oldIndexes) {
+		switch (oldIndexes.size()) {
+			case 0:
+				return Collections.emptyList();
+			case 1:
+				return previousParallelSubtaskStates.get(Iterables.getOnlyElement(oldIndexes));
+			default:
+				return oldIndexes.stream()
+					.flatMap(oldIndex -> previousParallelSubtaskStates.get(oldIndex).stream())
+					.collect(Collectors.toList());
+		}
+	}
+
+	protected Map<Integer, Set<Integer>> createNewToOldMapping(int oldParallelism, int newParallelism) {
+		return IntStream.range(0, newParallelism).boxed().
+			collect(Collectors.toMap(
+				Function.identity(),
+				channelIndex -> channelStateRescaler.getOldChannels(
+					channelIndex,
+					oldParallelism,
+					newParallelism)));
+	}
+
+	@Override
+	public List<List<T>> repartitionState(
+			List<List<T>> previousParallelSubtaskStates,
+			int oldParallelism,
+			int newParallelism) {
+		final Map<Integer, Set<Integer>> newToOldMapping = getNewToOldMapping(oldParallelism, newParallelism);
+		return IntStream.range(0, newParallelism)
+			.mapToObj(newIndex -> getOldState(previousParallelSubtaskStates, newToOldMapping.get(newIndex)))
+			.collect(Collectors.toList());

Review comment:
       (nit) I was a bit confused by this method. Maybe this version would be more readable:
   ```suggestion
   		return getNewToOldMapping(oldParallelism, newParallelism)
   			.entrySet().stream()
   			.sorted(Comparator.comparingInt(Map.Entry::getKey)).map(Map.Entry::getValue)
   			.map(oldIndices -> getOldState(previousParallelSubtaskStates, oldIndices))
   			.collect(Collectors.toList());
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e081fecba324a0e35017f4b89cc0499c9112a768 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517176519



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/GlobalPartitioner.java
##########
@@ -40,6 +41,16 @@ public int selectChannel(SerializationDelegate<StreamRecord<T>> record) {
 		return this;
 	}
 
+	@Override
+	public ChannelStateRescaler getUpstreamChannelStateRescaler() {
+		return ChannelStateRescaler.FIRST_CHANNEL;
+	}
+
+	@Override
+	public ChannelStateRescaler getDownstreamChannelStateRescaler() {
+		return ChannelStateRescaler.ROUND_ROBIN;

Review comment:
       Yes changed that (see comment below). On upstream it should be almost always round robin (broadcasting partitioner is the only exception where we drop extra state).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518179029



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/RescaledChannelsMapping.java
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import static java.util.Collections.emptyMap;
+
+/**
+ * Contains the fine-grain channel mappings that occur when a connected operator has been rescaled.
+ */
+public class RescaledChannelsMapping implements Serializable {
+	public static final RescaledChannelsMapping NO_CHANNEL_MAPPING = new RescaledChannelsMapping(emptyMap());
+
+	private static final long serialVersionUID = -8719670050630674631L;
+
+	/**
+	 * For each new channel (=index), all old channels are set.
+	 */
+	private final Map<Integer, Set<Integer>> newToOldChannelIndexes;
+
+	/**
+	 * For each old channel (=index), all new channels are set. Lazily calculated to keep {@link OperatorSubtaskState}
+	 * small in terms of serialization cost.
+	 */
+	private transient Map<Integer, Set<Integer>> oldToNewChannelIndexes;

Review comment:
       Actually I made it quasi-thread safe, where no fields are updated anymore (see your other suggestion). I refrain from doing defensive copies as it's purely internal.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714035151


   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit e081fecba324a0e35017f4b89cc0499c9112a768 (Thu Oct 22 00:12:43 UTC 2020)
   
   **Warnings:**
    * No documentation files were touched! Remember to keep the Flink docs up to date!
   
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518061766



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ChannelRescalerRepartitioner.java
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.io.network.api.writer.ChannelStateRescaler;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.Iterables;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * A repartitioner that assigns the same channel state to multiple subtasks according to some mapping.
+ *
+ * <p>The replicated data will then be filtered before processing the record.
+ *
+ * <p>Note that channel mappings are cached for the same parallelism changes.
+ */
+public class ChannelRescalerRepartitioner<T> implements OperatorStateRepartitioner<T> {
+	private final ChannelStateRescaler channelStateRescaler;
+	private final Map<Tuple2<Integer, Integer>, Map<Integer, Set<Integer>>> newToOldMappingCache = new HashMap<>(2);
+
+	public ChannelRescalerRepartitioner(ChannelStateRescaler channelStateRescaler) {
+		this.channelStateRescaler = channelStateRescaler;
+	}
+
+	private static <T> List<T> getOldState(List<List<T>> previousParallelSubtaskStates, Set<Integer> oldIndexes) {
+		switch (oldIndexes.size()) {
+			case 0:
+				return Collections.emptyList();
+			case 1:
+				return previousParallelSubtaskStates.get(Iterables.getOnlyElement(oldIndexes));
+			default:
+				return oldIndexes.stream()
+					.flatMap(oldIndex -> previousParallelSubtaskStates.get(oldIndex).stream())
+					.collect(Collectors.toList());
+		}
+	}
+
+	protected Map<Integer, Set<Integer>> createNewToOldMapping(int oldParallelism, int newParallelism) {
+		return IntStream.range(0, newParallelism).boxed().
+			collect(Collectors.toMap(
+				Function.identity(),
+				channelIndex -> channelStateRescaler.getOldChannels(
+					channelIndex,
+					oldParallelism,
+					newParallelism)));
+	}
+
+	@Override
+	public List<List<T>> repartitionState(
+			List<List<T>> previousParallelSubtaskStates,
+			int oldParallelism,
+			int newParallelism) {
+		final Map<Integer, Set<Integer>> newToOldMapping = getNewToOldMapping(oldParallelism, newParallelism);
+		return IntStream.range(0, newParallelism)
+			.mapToObj(newIndex -> getOldState(previousParallelSubtaskStates, newToOldMapping.get(newIndex)))
+			.collect(Collectors.toList());

Review comment:
       (nit) I was a bit confused by this method, I think it can be simplified like this:
   ```suggestion
   		return getNewToOldMapping(oldParallelism, newParallelism)
   			.values().stream()
   			.map(oldIndices -> getOldState(previousParallelSubtaskStates, oldIndices))
   			.collect(Collectors.toList());
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517395184



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
##########
@@ -99,32 +119,40 @@ public void assignStates() {
 						operatorID,
 						executionJobVertex.getParallelism(),
 						executionJobVertex.getMaxParallelism());
-				} else if (operatorState.getNumberCollectedStates() > 0) {
-					statelessSubTasks = false;
 				}
-				operatorStates.add(operatorState);
+				operatorStates.put(operatorIDPair.getGeneratedOperatorID(), operatorState);
 			}
-			if (!statelessSubTasks) { // skip tasks where no operator has any state
-				assignAttemptState(executionJobVertex, operatorStates);
+
+			final TaskStateAssignment stateAssignment = new TaskStateAssignment(executionJobVertex,	operatorStates);
+			vertexAssignments.put(executionJobVertex, stateAssignment);
+			for (final IntermediateResult producedDataSet : executionJobVertex.getInputs()) {
+				consumerAssignment.put(producedDataSet, stateAssignment);

Review comment:
       I see, thanks for the clarification and update.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518129564



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/AbstractChannelStateHandle.java
##########
@@ -46,6 +46,11 @@
 	private final List<Long> offsets;
 	private final long size;
 
+	/**
+	 * The original subtask index before rescaling recovery.
+	 */
+	private int originalSubtaskIndex;

Review comment:
       I made it part of the handle and it's fully serialized now. It's renamed to `subtaskIndex`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228",
       "triggerID" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8233",
       "triggerID" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8885",
       "triggerID" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 9cbbcb24751268c0b5289b86606eb28baa1287eb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8885) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517338838



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java
##########
@@ -725,6 +727,15 @@ public void deploy() throws JobException {
 			LOG.info("Deploying {} (attempt #{}) with attempt id {} to {} with allocation id {}", vertex.getTaskNameWithSubtaskIndex(),
 				attemptNumber, vertex.getCurrentExecutionAttempt().getAttemptId(), getAssignedResourceLocation(), slot.getAllocationId());
 
+			if (taskRestore != null) {
+				checkState(taskRestore.getTaskStateSnapshot().getSubtaskStateMappings().stream().allMatch(entry ->
+					entry.getValue().getInputChannelMappings().stream()
+						.allMatch(mapping -> mapping.equals(OperatorSubtaskState.VirtualChannelMapping.NO_MAPPING)) &&
+						entry.getValue().getOutputChannelMappings().stream()
+							.allMatch(mapping -> mapping.equals(OperatorSubtaskState.VirtualChannelMapping.NO_MAPPING))
+				), "Rescaling from unaligned checkpoint is not yet supported.");

Review comment:
       This is only temporary and could be avoided by having the second PR merged into the first PR. I needed a point to cancel while still allowing tests to run on the different pieces.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517389678



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -327,4 +289,252 @@ public boolean hasState() {
 			|| inputChannelState.hasState()
 			|| resultSubpartitionState.hasState();
 	}
+
+	public static Builder builder() {
+		return new Builder();
+	}
+
+	/**
+	 * The builder for a new {@link OperatorSubtaskState} which can be obtained by {@link #builder()}.
+	 */
+	public static class Builder {
+		private StateObjectCollection<OperatorStateHandle> managedOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<OperatorStateHandle> rawOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> managedKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> rawKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<InputChannelStateHandle> inputChannelState = StateObjectCollection.empty();
+		private StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState = StateObjectCollection.empty();
+		private VirtualChannelMapping inputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+		private VirtualChannelMapping outputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+
+		private Builder() {
+		}
+
+		public Builder setManagedOperatorState(StateObjectCollection<OperatorStateHandle> managedOperatorState) {
+			this.managedOperatorState = checkNotNull(managedOperatorState);
+			return this;
+		}
+
+		public Builder setManagedOperatorState(OperatorStateHandle managedOperatorState) {
+			return setManagedOperatorState(StateObjectCollection.singleton(checkNotNull(managedOperatorState)));
+		}
+
+		public Builder setRawOperatorState(StateObjectCollection<OperatorStateHandle> rawOperatorState) {
+			this.rawOperatorState = checkNotNull(rawOperatorState);
+			return this;
+		}
+
+		public Builder setRawOperatorState(OperatorStateHandle rawOperatorState) {
+			return setRawOperatorState(StateObjectCollection.singleton(checkNotNull(rawOperatorState)));
+		}
+
+		public Builder setManagedKeyedState(StateObjectCollection<KeyedStateHandle> managedKeyedState) {
+			this.managedKeyedState = checkNotNull(managedKeyedState);
+			return this;
+		}
+
+		public Builder setManagedKeyedState(KeyedStateHandle managedKeyedState) {
+			return setManagedKeyedState(StateObjectCollection.singleton(checkNotNull(managedKeyedState)));
+		}
+
+		public Builder setRawKeyedState(StateObjectCollection<KeyedStateHandle> rawKeyedState) {
+			this.rawKeyedState = checkNotNull(rawKeyedState);
+			return this;
+		}
+
+		public Builder setRawKeyedState(KeyedStateHandle rawKeyedState) {
+			return setRawKeyedState(StateObjectCollection.singleton(checkNotNull(rawKeyedState)));
+		}
+
+		public Builder setInputChannelState(StateObjectCollection<InputChannelStateHandle> inputChannelState) {
+			this.inputChannelState = checkNotNull(inputChannelState);
+			return this;
+		}
+
+		public Builder setResultSubpartitionState(StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState) {
+			this.resultSubpartitionState = checkNotNull(resultSubpartitionState);
+			return this;
+		}
+
+		public Builder setInputChannelMappings(VirtualChannelMapping inputChannelMappings) {
+			this.inputChannelMappings = checkNotNull(inputChannelMappings);
+			return this;
+		}
+
+		public Builder setOutputChannelMappings(VirtualChannelMapping outputChannelMappings) {
+			this.outputChannelMappings = checkNotNull(outputChannelMappings);
+			return this;
+		}
+
+		public OperatorSubtaskState build() {
+			return new OperatorSubtaskState(
+				managedOperatorState,
+				rawOperatorState,
+				managedKeyedState,
+				rawKeyedState,
+				inputChannelState,
+				resultSubpartitionState,
+				inputChannelMappings,
+				outputChannelMappings);
+		}
+	}
+
+	/**
+	 * Captures ambiguous mappings of old channels to new channels.
+	 *
+	 * <p>For inputs, this mapping implies the following:
+	 * <li>
+	 *     <ul>{@link #oldTaskInstances} is set when there is a rescale on this task potentially leading to different
+	 *     key groups. Upstream task has a corresponding {@link #partitionMappings} where it sends data over
+	 *     virtual channel while specifying the channel index in the VirtualChannelSelector. This subtask then
+	 *     demultiplexes over the virtual subtask index.</ul>
+	 *     <ul>{@link #partitionMappings} is set when there is a downscale of the upstream task. Upstream task has
+	 *     a corresponding {@link #oldTaskInstances} where it sends data over virtual channel while specifying the
+	 *     subtask index in the VirtualChannelSelector. This subtask then demultiplexes over channel indexes.</ul>
+	 * </li>
+	 *
+	 * <p>For outputs, it's vice-versa. The information must be kept in sync but they are used in opposite ways for
+	 * multiplexing/demultiplexing.
+	 *
+	 * <p>Note that in the common rescaling case both information is set and need to be simultaneously used. If the
+	 * input subtask subsumes the state of 3 old subtasks and a channel corresponds to 2 old channels, then there are
+	 * 6 virtual channels to be demultiplexed.
+	 */
+	public static class VirtualChannelMapping implements Serializable {

Review comment:
       I think `ChannelStateMapping` wouldn't be consistent with a field:
   ```
   /**
    * Set when several operator instances are merged into one.
    */
   private final BitSet oldTaskInstances;
   ```
   
   Maybe something like `EdgeRescalingDescriptor`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517717510



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
##########
@@ -620,18 +758,112 @@ private static void checkStateMappingCompleteness(
 			chainOpParallelStates,
 			oldParallelism,
 			newParallelism);
+	}
+
+	static class TaskStateAssignment {

Review comment:
       I split the two pass commit into two, but they are strongly related, so I'm not sure if the should exist on their own.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518152519



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ChannelRescalerRepartitioner.java
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.io.network.api.writer.ChannelStateRescaler;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.Iterables;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * A repartitioner that assigns the same channel state to multiple subtasks according to some mapping.
+ *
+ * <p>The replicated data will then be filtered before processing the record.
+ *
+ * <p>Note that channel mappings are cached for the same parallelism changes.
+ */
+public class ChannelRescalerRepartitioner<T> implements OperatorStateRepartitioner<T> {
+	private final ChannelStateRescaler channelStateRescaler;
+	private final Map<Tuple2<Integer, Integer>, Map<Integer, Set<Integer>>> newToOldMappingCache = new HashMap<>(2);

Review comment:
       You are right. I needed to adjust abstractions a bit (and I think it's more logical now):
   - `createNewToOldMapping` is now part of `ChannelStateRescaler` (which is now named `SubtaskStateMapper`).
   - `ChannelRescalerRepartitioner` is renamed to `MappingBasedRepartitioner`. It's an pseudo-immutable class that simply applies the mapping to the state. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517348994



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -327,4 +289,252 @@ public boolean hasState() {
 			|| inputChannelState.hasState()
 			|| resultSubpartitionState.hasState();
 	}
+
+	public static Builder builder() {
+		return new Builder();
+	}
+
+	/**
+	 * The builder for a new {@link OperatorSubtaskState} which can be obtained by {@link #builder()}.
+	 */
+	public static class Builder {
+		private StateObjectCollection<OperatorStateHandle> managedOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<OperatorStateHandle> rawOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> managedKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> rawKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<InputChannelStateHandle> inputChannelState = StateObjectCollection.empty();
+		private StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState = StateObjectCollection.empty();
+		private VirtualChannelMapping inputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+		private VirtualChannelMapping outputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+
+		private Builder() {
+		}
+
+		public Builder setManagedOperatorState(StateObjectCollection<OperatorStateHandle> managedOperatorState) {
+			this.managedOperatorState = checkNotNull(managedOperatorState);
+			return this;
+		}
+
+		public Builder setManagedOperatorState(OperatorStateHandle managedOperatorState) {
+			return setManagedOperatorState(StateObjectCollection.singleton(checkNotNull(managedOperatorState)));
+		}
+
+		public Builder setRawOperatorState(StateObjectCollection<OperatorStateHandle> rawOperatorState) {
+			this.rawOperatorState = checkNotNull(rawOperatorState);
+			return this;
+		}
+
+		public Builder setRawOperatorState(OperatorStateHandle rawOperatorState) {
+			return setRawOperatorState(StateObjectCollection.singleton(checkNotNull(rawOperatorState)));
+		}
+
+		public Builder setManagedKeyedState(StateObjectCollection<KeyedStateHandle> managedKeyedState) {
+			this.managedKeyedState = checkNotNull(managedKeyedState);
+			return this;
+		}
+
+		public Builder setManagedKeyedState(KeyedStateHandle managedKeyedState) {
+			return setManagedKeyedState(StateObjectCollection.singleton(checkNotNull(managedKeyedState)));
+		}
+
+		public Builder setRawKeyedState(StateObjectCollection<KeyedStateHandle> rawKeyedState) {
+			this.rawKeyedState = checkNotNull(rawKeyedState);
+			return this;
+		}
+
+		public Builder setRawKeyedState(KeyedStateHandle rawKeyedState) {
+			return setRawKeyedState(StateObjectCollection.singleton(checkNotNull(rawKeyedState)));
+		}
+
+		public Builder setInputChannelState(StateObjectCollection<InputChannelStateHandle> inputChannelState) {
+			this.inputChannelState = checkNotNull(inputChannelState);
+			return this;
+		}
+
+		public Builder setResultSubpartitionState(StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState) {
+			this.resultSubpartitionState = checkNotNull(resultSubpartitionState);
+			return this;
+		}
+
+		public Builder setInputChannelMappings(VirtualChannelMapping inputChannelMappings) {
+			this.inputChannelMappings = checkNotNull(inputChannelMappings);
+			return this;
+		}
+
+		public Builder setOutputChannelMappings(VirtualChannelMapping outputChannelMappings) {
+			this.outputChannelMappings = checkNotNull(outputChannelMappings);
+			return this;
+		}
+
+		public OperatorSubtaskState build() {
+			return new OperatorSubtaskState(
+				managedOperatorState,
+				rawOperatorState,
+				managedKeyedState,
+				rawKeyedState,
+				inputChannelState,
+				resultSubpartitionState,
+				inputChannelMappings,
+				outputChannelMappings);
+		}
+	}
+
+	/**
+	 * Captures ambiguous mappings of old channels to new channels.
+	 *
+	 * <p>For inputs, this mapping implies the following:
+	 * <li>
+	 *     <ul>{@link #oldTaskInstances} is set when there is a rescale on this task potentially leading to different
+	 *     key groups. Upstream task has a corresponding {@link #partitionMappings} where it sends data over
+	 *     virtual channel while specifying the channel index in the VirtualChannelSelector. This subtask then
+	 *     demultiplexes over the virtual subtask index.</ul>
+	 *     <ul>{@link #partitionMappings} is set when there is a downscale of the upstream task. Upstream task has
+	 *     a corresponding {@link #oldTaskInstances} where it sends data over virtual channel while specifying the
+	 *     subtask index in the VirtualChannelSelector. This subtask then demultiplexes over channel indexes.</ul>
+	 * </li>
+	 *
+	 * <p>For outputs, it's vice-versa. The information must be kept in sync but they are used in opposite ways for
+	 * multiplexing/demultiplexing.
+	 *
+	 * <p>Note that in the common rescaling case both information is set and need to be simultaneously used. If the
+	 * input subtask subsumes the state of 3 old subtasks and a channel corresponds to 2 old channels, then there are
+	 * 6 virtual channels to be demultiplexed.
+	 */
+	public static class VirtualChannelMapping implements Serializable {
+		public static final PartitionMapping NO_CHANNEL_MAPPING = new PartitionMapping(emptyList());
+		public static final List<PartitionMapping> NO_PARTITIONS = emptyList();
+		public static final BitSet NO_SUBTASKS = new BitSet();
+		public static final VirtualChannelMapping NO_MAPPING = new VirtualChannelMapping(NO_SUBTASKS, NO_PARTITIONS);
+
+		/**
+		 * Set when several operator instances are merged into one.
+		 */
+		private final BitSet oldTaskInstances;
+
+		/**
+		 * Set when channels are merged because the connected operator has been rescaled.
+		 */
+		private final List<PartitionMapping> partitionMappings;
+
+		public VirtualChannelMapping(BitSet oldTaskInstances, List<PartitionMapping> partitionMappings) {
+			this.oldTaskInstances = oldTaskInstances;
+			this.partitionMappings = partitionMappings;
+		}
+
+		@Override
+		public boolean equals(Object o) {
+			if (this == o) {
+				return true;
+			}
+			if (o == null || getClass() != o.getClass()) {
+				return false;
+			}
+			final VirtualChannelMapping that = (VirtualChannelMapping) o;
+			return oldTaskInstances.equals(that.oldTaskInstances) &&
+				partitionMappings.equals(that.partitionMappings);
+		}
+
+		public int[] getOldTaskInstances(int defaultSubtask) {
+			return oldTaskInstances.equals(NO_SUBTASKS) ?
+				new int[] {defaultSubtask} :
+				oldTaskInstances.stream().toArray();
+		}
+
+		public PartitionMapping getPartitionMapping(int partitionIndex) {
+			if (partitionMappings.isEmpty()) {
+				return NO_CHANNEL_MAPPING;
+			}
+			return partitionMappings.get(partitionIndex);
+		}
+
+		@Override
+		public int hashCode() {
+			return Objects.hash(oldTaskInstances, partitionMappings);
+		}
+
+		@Override
+		public String toString() {
+			return "VirtualChannelMapping{" +
+				"oldTaskInstances=" + oldTaskInstances +
+				", partitionMappings=" + partitionMappings +
+				'}';
+		}
+	}
+
+	/**
+	 * Contains the fine-grain channel mappings that occur when a connected operator has been rescaled.
+	 */
+	public static class PartitionMapping implements Serializable {
+
+		/**
+		 * For each new channel (=index), all old channels are set.
+		 */
+		private final List<BitSet> newToOldChannelIndexes;

Review comment:
       If we have consecutive indexes from 0..n, what's wrong with `List<Set<Integer>>`? (I got the BitSet part)
   btw it would be `Map<Int, Set<Int>>`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r516902951



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -327,4 +289,252 @@ public boolean hasState() {
 			|| inputChannelState.hasState()
 			|| resultSubpartitionState.hasState();
 	}
+
+	public static Builder builder() {
+		return new Builder();
+	}
+
+	/**
+	 * The builder for a new {@link OperatorSubtaskState} which can be obtained by {@link #builder()}.
+	 */
+	public static class Builder {
+		private StateObjectCollection<OperatorStateHandle> managedOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<OperatorStateHandle> rawOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> managedKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> rawKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<InputChannelStateHandle> inputChannelState = StateObjectCollection.empty();
+		private StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState = StateObjectCollection.empty();
+		private VirtualChannelMapping inputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+		private VirtualChannelMapping outputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+
+		private Builder() {
+		}
+
+		public Builder setManagedOperatorState(StateObjectCollection<OperatorStateHandle> managedOperatorState) {
+			this.managedOperatorState = checkNotNull(managedOperatorState);
+			return this;
+		}
+
+		public Builder setManagedOperatorState(OperatorStateHandle managedOperatorState) {
+			return setManagedOperatorState(StateObjectCollection.singleton(checkNotNull(managedOperatorState)));
+		}
+
+		public Builder setRawOperatorState(StateObjectCollection<OperatorStateHandle> rawOperatorState) {
+			this.rawOperatorState = checkNotNull(rawOperatorState);
+			return this;
+		}
+
+		public Builder setRawOperatorState(OperatorStateHandle rawOperatorState) {
+			return setRawOperatorState(StateObjectCollection.singleton(checkNotNull(rawOperatorState)));
+		}
+
+		public Builder setManagedKeyedState(StateObjectCollection<KeyedStateHandle> managedKeyedState) {
+			this.managedKeyedState = checkNotNull(managedKeyedState);
+			return this;
+		}
+
+		public Builder setManagedKeyedState(KeyedStateHandle managedKeyedState) {
+			return setManagedKeyedState(StateObjectCollection.singleton(checkNotNull(managedKeyedState)));
+		}
+
+		public Builder setRawKeyedState(StateObjectCollection<KeyedStateHandle> rawKeyedState) {
+			this.rawKeyedState = checkNotNull(rawKeyedState);
+			return this;
+		}
+
+		public Builder setRawKeyedState(KeyedStateHandle rawKeyedState) {
+			return setRawKeyedState(StateObjectCollection.singleton(checkNotNull(rawKeyedState)));
+		}
+
+		public Builder setInputChannelState(StateObjectCollection<InputChannelStateHandle> inputChannelState) {
+			this.inputChannelState = checkNotNull(inputChannelState);
+			return this;
+		}
+
+		public Builder setResultSubpartitionState(StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState) {
+			this.resultSubpartitionState = checkNotNull(resultSubpartitionState);
+			return this;
+		}
+
+		public Builder setInputChannelMappings(VirtualChannelMapping inputChannelMappings) {
+			this.inputChannelMappings = checkNotNull(inputChannelMappings);
+			return this;
+		}
+
+		public Builder setOutputChannelMappings(VirtualChannelMapping outputChannelMappings) {
+			this.outputChannelMappings = checkNotNull(outputChannelMappings);
+			return this;
+		}
+
+		public OperatorSubtaskState build() {
+			return new OperatorSubtaskState(
+				managedOperatorState,
+				rawOperatorState,
+				managedKeyedState,
+				rawKeyedState,
+				inputChannelState,
+				resultSubpartitionState,
+				inputChannelMappings,
+				outputChannelMappings);
+		}
+	}
+
+	/**
+	 * Captures ambiguous mappings of old channels to new channels.
+	 *
+	 * <p>For inputs, this mapping implies the following:
+	 * <li>
+	 *     <ul>{@link #oldTaskInstances} is set when there is a rescale on this task potentially leading to different
+	 *     key groups. Upstream task has a corresponding {@link #partitionMappings} where it sends data over
+	 *     virtual channel while specifying the channel index in the VirtualChannelSelector. This subtask then
+	 *     demultiplexes over the virtual subtask index.</ul>
+	 *     <ul>{@link #partitionMappings} is set when there is a downscale of the upstream task. Upstream task has
+	 *     a corresponding {@link #oldTaskInstances} where it sends data over virtual channel while specifying the
+	 *     subtask index in the VirtualChannelSelector. This subtask then demultiplexes over channel indexes.</ul>
+	 * </li>
+	 *
+	 * <p>For outputs, it's vice-versa. The information must be kept in sync but they are used in opposite ways for
+	 * multiplexing/demultiplexing.
+	 *
+	 * <p>Note that in the common rescaling case both information is set and need to be simultaneously used. If the
+	 * input subtask subsumes the state of 3 old subtasks and a channel corresponds to 2 old channels, then there are
+	 * 6 virtual channels to be demultiplexed.
+	 */
+	public static class VirtualChannelMapping implements Serializable {

Review comment:
       1. Is it actually **channel** mapping? I think it actually remaps operator state, doesn't it?
   2. `serialVersionUID`?
   3. I'd rather move this class to a separate file

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -327,4 +289,252 @@ public boolean hasState() {
 			|| inputChannelState.hasState()
 			|| resultSubpartitionState.hasState();
 	}
+
+	public static Builder builder() {
+		return new Builder();
+	}
+
+	/**
+	 * The builder for a new {@link OperatorSubtaskState} which can be obtained by {@link #builder()}.
+	 */
+	public static class Builder {
+		private StateObjectCollection<OperatorStateHandle> managedOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<OperatorStateHandle> rawOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> managedKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> rawKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<InputChannelStateHandle> inputChannelState = StateObjectCollection.empty();
+		private StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState = StateObjectCollection.empty();
+		private VirtualChannelMapping inputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+		private VirtualChannelMapping outputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+
+		private Builder() {
+		}
+
+		public Builder setManagedOperatorState(StateObjectCollection<OperatorStateHandle> managedOperatorState) {
+			this.managedOperatorState = checkNotNull(managedOperatorState);
+			return this;
+		}
+
+		public Builder setManagedOperatorState(OperatorStateHandle managedOperatorState) {
+			return setManagedOperatorState(StateObjectCollection.singleton(checkNotNull(managedOperatorState)));
+		}
+
+		public Builder setRawOperatorState(StateObjectCollection<OperatorStateHandle> rawOperatorState) {
+			this.rawOperatorState = checkNotNull(rawOperatorState);
+			return this;
+		}
+
+		public Builder setRawOperatorState(OperatorStateHandle rawOperatorState) {
+			return setRawOperatorState(StateObjectCollection.singleton(checkNotNull(rawOperatorState)));
+		}
+
+		public Builder setManagedKeyedState(StateObjectCollection<KeyedStateHandle> managedKeyedState) {
+			this.managedKeyedState = checkNotNull(managedKeyedState);
+			return this;
+		}
+
+		public Builder setManagedKeyedState(KeyedStateHandle managedKeyedState) {
+			return setManagedKeyedState(StateObjectCollection.singleton(checkNotNull(managedKeyedState)));
+		}
+
+		public Builder setRawKeyedState(StateObjectCollection<KeyedStateHandle> rawKeyedState) {
+			this.rawKeyedState = checkNotNull(rawKeyedState);
+			return this;
+		}
+
+		public Builder setRawKeyedState(KeyedStateHandle rawKeyedState) {
+			return setRawKeyedState(StateObjectCollection.singleton(checkNotNull(rawKeyedState)));
+		}
+
+		public Builder setInputChannelState(StateObjectCollection<InputChannelStateHandle> inputChannelState) {
+			this.inputChannelState = checkNotNull(inputChannelState);
+			return this;
+		}
+
+		public Builder setResultSubpartitionState(StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState) {
+			this.resultSubpartitionState = checkNotNull(resultSubpartitionState);
+			return this;
+		}
+
+		public Builder setInputChannelMappings(VirtualChannelMapping inputChannelMappings) {
+			this.inputChannelMappings = checkNotNull(inputChannelMappings);
+			return this;
+		}
+
+		public Builder setOutputChannelMappings(VirtualChannelMapping outputChannelMappings) {
+			this.outputChannelMappings = checkNotNull(outputChannelMappings);
+			return this;
+		}
+
+		public OperatorSubtaskState build() {
+			return new OperatorSubtaskState(
+				managedOperatorState,
+				rawOperatorState,
+				managedKeyedState,
+				rawKeyedState,
+				inputChannelState,
+				resultSubpartitionState,
+				inputChannelMappings,
+				outputChannelMappings);
+		}
+	}
+
+	/**
+	 * Captures ambiguous mappings of old channels to new channels.
+	 *
+	 * <p>For inputs, this mapping implies the following:
+	 * <li>
+	 *     <ul>{@link #oldTaskInstances} is set when there is a rescale on this task potentially leading to different
+	 *     key groups. Upstream task has a corresponding {@link #partitionMappings} where it sends data over
+	 *     virtual channel while specifying the channel index in the VirtualChannelSelector. This subtask then
+	 *     demultiplexes over the virtual subtask index.</ul>
+	 *     <ul>{@link #partitionMappings} is set when there is a downscale of the upstream task. Upstream task has
+	 *     a corresponding {@link #oldTaskInstances} where it sends data over virtual channel while specifying the
+	 *     subtask index in the VirtualChannelSelector. This subtask then demultiplexes over channel indexes.</ul>
+	 * </li>
+	 *
+	 * <p>For outputs, it's vice-versa. The information must be kept in sync but they are used in opposite ways for
+	 * multiplexing/demultiplexing.
+	 *
+	 * <p>Note that in the common rescaling case both information is set and need to be simultaneously used. If the
+	 * input subtask subsumes the state of 3 old subtasks and a channel corresponds to 2 old channels, then there are
+	 * 6 virtual channels to be demultiplexed.
+	 */
+	public static class VirtualChannelMapping implements Serializable {
+		public static final PartitionMapping NO_CHANNEL_MAPPING = new PartitionMapping(emptyList());
+		public static final List<PartitionMapping> NO_PARTITIONS = emptyList();
+		public static final BitSet NO_SUBTASKS = new BitSet();
+		public static final VirtualChannelMapping NO_MAPPING = new VirtualChannelMapping(NO_SUBTASKS, NO_PARTITIONS);
+
+		/**
+		 * Set when several operator instances are merged into one.
+		 */
+		private final BitSet oldTaskInstances;
+
+		/**
+		 * Set when channels are merged because the connected operator has been rescaled.
+		 */
+		private final List<PartitionMapping> partitionMappings;

Review comment:
       To me, list isn't an obvious choice to provide index-based access.
   Array or hashtable would be more readable to me and guarantee O(1) time.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
##########
@@ -99,32 +119,40 @@ public void assignStates() {
 						operatorID,
 						executionJobVertex.getParallelism(),
 						executionJobVertex.getMaxParallelism());
-				} else if (operatorState.getNumberCollectedStates() > 0) {
-					statelessSubTasks = false;
 				}
-				operatorStates.add(operatorState);
+				operatorStates.put(operatorIDPair.getGeneratedOperatorID(), operatorState);
 			}
-			if (!statelessSubTasks) { // skip tasks where no operator has any state
-				assignAttemptState(executionJobVertex, operatorStates);
+
+			final TaskStateAssignment stateAssignment = new TaskStateAssignment(executionJobVertex,	operatorStates);
+			vertexAssignments.put(executionJobVertex, stateAssignment);
+			for (final IntermediateResult producedDataSet : executionJobVertex.getInputs()) {
+				consumerAssignment.put(producedDataSet, stateAssignment);

Review comment:
       1. I wonder if this `put` can override existing `assignment` (e.g. with a `UnionGate`).
   2. `IntermediateResult` use as a key doesn't override equals/hashCode - is it intentional?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/ForwardPartitioner.java
##########
@@ -43,4 +44,14 @@ public int selectChannel(SerializationDelegate<StreamRecord<T>> record) {
 	public String toString() {
 		return "FORWARD";
 	}
+
+	@Override
+	public ChannelStateRescaler getUpstreamChannelStateRescaler() {
+		return ChannelStateRescaler.FIRST_CHANNEL;
+	}
+
+	@Override
+	public ChannelStateRescaler getDownstreamChannelStateRescaler() {
+		return ChannelStateRescaler.ROUND_ROBIN;

Review comment:
       Why isn't channel state distributed to the same subtask as before (or 1st if out of range)?
   (ditto upstream)
   Shouldn't recovered buffers sent from the upstream match what is on the downstream?

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/GlobalPartitioner.java
##########
@@ -40,6 +41,16 @@ public int selectChannel(SerializationDelegate<StreamRecord<T>> record) {
 		return this;
 	}
 
+	@Override
+	public ChannelStateRescaler getUpstreamChannelStateRescaler() {
+		return ChannelStateRescaler.FIRST_CHANNEL;
+	}
+
+	@Override
+	public ChannelStateRescaler getDownstreamChannelStateRescaler() {
+		return ChannelStateRescaler.ROUND_ROBIN;

Review comment:
       I'm probably misunderstanding it, but it seems it should be the opposite:
   upstream: `ROUND_ROBING` (or both FIRST)
   downstream: `FIRST_CHANNEL`
   

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
##########
@@ -620,18 +758,112 @@ private static void checkStateMappingCompleteness(
 			chainOpParallelStates,
 			oldParallelism,
 			newParallelism);
+	}
+
+	static class TaskStateAssignment {

Review comment:
       1. It would ease review if this class was introduced in a separate refactoring commit (though it's probably too difficult at this stage)
   2. Can we extract this class to a separate file?

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
##########
@@ -67,29 +83,33 @@
 	private final long restoreCheckpointId;
 	private final boolean allowNonRestoredState;
 
+	private final Map<IntermediateResult, TaskStateAssignment> consumerAssignment = new HashMap<>();
+	private final Map<ChannelStateRescaler, ChannelRescalerRepartitioner<Object>> rescalerRepartitioners =

Review comment:
       IIUC, this is a cache, right?
   I doubt that we really need it: it's only used on re/starts and can save maybe hundreds of ms with DOP=1K. And without it, the code would be much simpler.
   WDYT?

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java
##########
@@ -725,6 +727,15 @@ public void deploy() throws JobException {
 			LOG.info("Deploying {} (attempt #{}) with attempt id {} to {} with allocation id {}", vertex.getTaskNameWithSubtaskIndex(),
 				attemptNumber, vertex.getCurrentExecutionAttempt().getAttemptId(), getAssignedResourceLocation(), slot.getAllocationId());
 
+			if (taskRestore != null) {
+				checkState(taskRestore.getTaskStateSnapshot().getSubtaskStateMappings().stream().allMatch(entry ->
+					entry.getValue().getInputChannelMappings().stream()
+						.allMatch(mapping -> mapping.equals(OperatorSubtaskState.VirtualChannelMapping.NO_MAPPING)) &&
+						entry.getValue().getOutputChannelMappings().stream()
+							.allMatch(mapping -> mapping.equals(OperatorSubtaskState.VirtualChannelMapping.NO_MAPPING))
+				), "Rescaling from unaligned checkpoint is not yet supported.");

Review comment:
       1. This doesn't compile (cannot find symbol `getInputChannelMappings`)
   2. Probably makes sense to extract testing methods (unless it's a temporary method)

##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java
##########
@@ -692,6 +692,8 @@ private void connect(Integer headOfChain, StreamEdge edge) {
 		}
 		// set strategy name so that web interface can show it.
 		jobEdge.setShipStrategyName(partitioner.toString());
+		jobEdge.setDownstreamChannelStateRescaler(partitioner.getUpstreamChannelStateRescaler());
+		jobEdge.setUpstreamChannelStateRescaler(partitioner.getDownstreamChannelStateRescaler());

Review comment:
       OK now I see why the Rescalers used seemed inverted to me :)
   So why is it inverted here? (maybe add a comment?)

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -327,4 +289,252 @@ public boolean hasState() {
 			|| inputChannelState.hasState()
 			|| resultSubpartitionState.hasState();
 	}
+
+	public static Builder builder() {
+		return new Builder();
+	}
+
+	/**
+	 * The builder for a new {@link OperatorSubtaskState} which can be obtained by {@link #builder()}.
+	 */
+	public static class Builder {
+		private StateObjectCollection<OperatorStateHandle> managedOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<OperatorStateHandle> rawOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> managedKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> rawKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<InputChannelStateHandle> inputChannelState = StateObjectCollection.empty();
+		private StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState = StateObjectCollection.empty();
+		private VirtualChannelMapping inputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+		private VirtualChannelMapping outputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+
+		private Builder() {
+		}
+
+		public Builder setManagedOperatorState(StateObjectCollection<OperatorStateHandle> managedOperatorState) {
+			this.managedOperatorState = checkNotNull(managedOperatorState);
+			return this;
+		}
+
+		public Builder setManagedOperatorState(OperatorStateHandle managedOperatorState) {
+			return setManagedOperatorState(StateObjectCollection.singleton(checkNotNull(managedOperatorState)));
+		}
+
+		public Builder setRawOperatorState(StateObjectCollection<OperatorStateHandle> rawOperatorState) {
+			this.rawOperatorState = checkNotNull(rawOperatorState);
+			return this;
+		}
+
+		public Builder setRawOperatorState(OperatorStateHandle rawOperatorState) {
+			return setRawOperatorState(StateObjectCollection.singleton(checkNotNull(rawOperatorState)));
+		}
+
+		public Builder setManagedKeyedState(StateObjectCollection<KeyedStateHandle> managedKeyedState) {
+			this.managedKeyedState = checkNotNull(managedKeyedState);
+			return this;
+		}
+
+		public Builder setManagedKeyedState(KeyedStateHandle managedKeyedState) {
+			return setManagedKeyedState(StateObjectCollection.singleton(checkNotNull(managedKeyedState)));
+		}
+
+		public Builder setRawKeyedState(StateObjectCollection<KeyedStateHandle> rawKeyedState) {
+			this.rawKeyedState = checkNotNull(rawKeyedState);
+			return this;
+		}
+
+		public Builder setRawKeyedState(KeyedStateHandle rawKeyedState) {
+			return setRawKeyedState(StateObjectCollection.singleton(checkNotNull(rawKeyedState)));
+		}
+
+		public Builder setInputChannelState(StateObjectCollection<InputChannelStateHandle> inputChannelState) {
+			this.inputChannelState = checkNotNull(inputChannelState);
+			return this;
+		}
+
+		public Builder setResultSubpartitionState(StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState) {
+			this.resultSubpartitionState = checkNotNull(resultSubpartitionState);
+			return this;
+		}
+
+		public Builder setInputChannelMappings(VirtualChannelMapping inputChannelMappings) {
+			this.inputChannelMappings = checkNotNull(inputChannelMappings);
+			return this;
+		}
+
+		public Builder setOutputChannelMappings(VirtualChannelMapping outputChannelMappings) {
+			this.outputChannelMappings = checkNotNull(outputChannelMappings);
+			return this;
+		}
+
+		public OperatorSubtaskState build() {
+			return new OperatorSubtaskState(
+				managedOperatorState,
+				rawOperatorState,
+				managedKeyedState,
+				rawKeyedState,
+				inputChannelState,
+				resultSubpartitionState,
+				inputChannelMappings,
+				outputChannelMappings);
+		}
+	}
+
+	/**
+	 * Captures ambiguous mappings of old channels to new channels.
+	 *
+	 * <p>For inputs, this mapping implies the following:
+	 * <li>
+	 *     <ul>{@link #oldTaskInstances} is set when there is a rescale on this task potentially leading to different
+	 *     key groups. Upstream task has a corresponding {@link #partitionMappings} where it sends data over
+	 *     virtual channel while specifying the channel index in the VirtualChannelSelector. This subtask then
+	 *     demultiplexes over the virtual subtask index.</ul>
+	 *     <ul>{@link #partitionMappings} is set when there is a downscale of the upstream task. Upstream task has
+	 *     a corresponding {@link #oldTaskInstances} where it sends data over virtual channel while specifying the
+	 *     subtask index in the VirtualChannelSelector. This subtask then demultiplexes over channel indexes.</ul>
+	 * </li>
+	 *
+	 * <p>For outputs, it's vice-versa. The information must be kept in sync but they are used in opposite ways for
+	 * multiplexing/demultiplexing.
+	 *
+	 * <p>Note that in the common rescaling case both information is set and need to be simultaneously used. If the
+	 * input subtask subsumes the state of 3 old subtasks and a channel corresponds to 2 old channels, then there are
+	 * 6 virtual channels to be demultiplexed.
+	 */
+	public static class VirtualChannelMapping implements Serializable {
+		public static final PartitionMapping NO_CHANNEL_MAPPING = new PartitionMapping(emptyList());
+		public static final List<PartitionMapping> NO_PARTITIONS = emptyList();
+		public static final BitSet NO_SUBTASKS = new BitSet();
+		public static final VirtualChannelMapping NO_MAPPING = new VirtualChannelMapping(NO_SUBTASKS, NO_PARTITIONS);
+
+		/**
+		 * Set when several operator instances are merged into one.
+		 */
+		private final BitSet oldTaskInstances;

Review comment:
       Why bitsets are used here and throughout the PR?
   I think just `Set<Integer>` would be
   a) more readable
   b) more efficient (no need for extra words if only high bits are sets)

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -327,4 +289,252 @@ public boolean hasState() {
 			|| inputChannelState.hasState()
 			|| resultSubpartitionState.hasState();
 	}
+
+	public static Builder builder() {
+		return new Builder();
+	}
+
+	/**
+	 * The builder for a new {@link OperatorSubtaskState} which can be obtained by {@link #builder()}.
+	 */
+	public static class Builder {
+		private StateObjectCollection<OperatorStateHandle> managedOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<OperatorStateHandle> rawOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> managedKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> rawKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<InputChannelStateHandle> inputChannelState = StateObjectCollection.empty();
+		private StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState = StateObjectCollection.empty();
+		private VirtualChannelMapping inputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+		private VirtualChannelMapping outputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+
+		private Builder() {
+		}
+
+		public Builder setManagedOperatorState(StateObjectCollection<OperatorStateHandle> managedOperatorState) {
+			this.managedOperatorState = checkNotNull(managedOperatorState);
+			return this;
+		}
+
+		public Builder setManagedOperatorState(OperatorStateHandle managedOperatorState) {
+			return setManagedOperatorState(StateObjectCollection.singleton(checkNotNull(managedOperatorState)));
+		}
+
+		public Builder setRawOperatorState(StateObjectCollection<OperatorStateHandle> rawOperatorState) {
+			this.rawOperatorState = checkNotNull(rawOperatorState);
+			return this;
+		}
+
+		public Builder setRawOperatorState(OperatorStateHandle rawOperatorState) {
+			return setRawOperatorState(StateObjectCollection.singleton(checkNotNull(rawOperatorState)));
+		}
+
+		public Builder setManagedKeyedState(StateObjectCollection<KeyedStateHandle> managedKeyedState) {
+			this.managedKeyedState = checkNotNull(managedKeyedState);
+			return this;
+		}
+
+		public Builder setManagedKeyedState(KeyedStateHandle managedKeyedState) {
+			return setManagedKeyedState(StateObjectCollection.singleton(checkNotNull(managedKeyedState)));
+		}
+
+		public Builder setRawKeyedState(StateObjectCollection<KeyedStateHandle> rawKeyedState) {
+			this.rawKeyedState = checkNotNull(rawKeyedState);
+			return this;
+		}
+
+		public Builder setRawKeyedState(KeyedStateHandle rawKeyedState) {
+			return setRawKeyedState(StateObjectCollection.singleton(checkNotNull(rawKeyedState)));
+		}
+
+		public Builder setInputChannelState(StateObjectCollection<InputChannelStateHandle> inputChannelState) {
+			this.inputChannelState = checkNotNull(inputChannelState);
+			return this;
+		}
+
+		public Builder setResultSubpartitionState(StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState) {
+			this.resultSubpartitionState = checkNotNull(resultSubpartitionState);
+			return this;
+		}
+
+		public Builder setInputChannelMappings(VirtualChannelMapping inputChannelMappings) {
+			this.inputChannelMappings = checkNotNull(inputChannelMappings);
+			return this;
+		}
+
+		public Builder setOutputChannelMappings(VirtualChannelMapping outputChannelMappings) {
+			this.outputChannelMappings = checkNotNull(outputChannelMappings);
+			return this;
+		}
+
+		public OperatorSubtaskState build() {
+			return new OperatorSubtaskState(
+				managedOperatorState,
+				rawOperatorState,
+				managedKeyedState,
+				rawKeyedState,
+				inputChannelState,
+				resultSubpartitionState,
+				inputChannelMappings,
+				outputChannelMappings);
+		}
+	}
+
+	/**
+	 * Captures ambiguous mappings of old channels to new channels.
+	 *
+	 * <p>For inputs, this mapping implies the following:
+	 * <li>
+	 *     <ul>{@link #oldTaskInstances} is set when there is a rescale on this task potentially leading to different
+	 *     key groups. Upstream task has a corresponding {@link #partitionMappings} where it sends data over
+	 *     virtual channel while specifying the channel index in the VirtualChannelSelector. This subtask then
+	 *     demultiplexes over the virtual subtask index.</ul>
+	 *     <ul>{@link #partitionMappings} is set when there is a downscale of the upstream task. Upstream task has
+	 *     a corresponding {@link #oldTaskInstances} where it sends data over virtual channel while specifying the
+	 *     subtask index in the VirtualChannelSelector. This subtask then demultiplexes over channel indexes.</ul>
+	 * </li>
+	 *
+	 * <p>For outputs, it's vice-versa. The information must be kept in sync but they are used in opposite ways for
+	 * multiplexing/demultiplexing.
+	 *
+	 * <p>Note that in the common rescaling case both information is set and need to be simultaneously used. If the
+	 * input subtask subsumes the state of 3 old subtasks and a channel corresponds to 2 old channels, then there are
+	 * 6 virtual channels to be demultiplexed.
+	 */
+	public static class VirtualChannelMapping implements Serializable {
+		public static final PartitionMapping NO_CHANNEL_MAPPING = new PartitionMapping(emptyList());
+		public static final List<PartitionMapping> NO_PARTITIONS = emptyList();
+		public static final BitSet NO_SUBTASKS = new BitSet();
+		public static final VirtualChannelMapping NO_MAPPING = new VirtualChannelMapping(NO_SUBTASKS, NO_PARTITIONS);
+
+		/**
+		 * Set when several operator instances are merged into one.
+		 */
+		private final BitSet oldTaskInstances;
+
+		/**
+		 * Set when channels are merged because the connected operator has been rescaled.
+		 */
+		private final List<PartitionMapping> partitionMappings;
+
+		public VirtualChannelMapping(BitSet oldTaskInstances, List<PartitionMapping> partitionMappings) {
+			this.oldTaskInstances = oldTaskInstances;
+			this.partitionMappings = partitionMappings;
+		}
+
+		@Override
+		public boolean equals(Object o) {
+			if (this == o) {
+				return true;
+			}
+			if (o == null || getClass() != o.getClass()) {
+				return false;
+			}
+			final VirtualChannelMapping that = (VirtualChannelMapping) o;
+			return oldTaskInstances.equals(that.oldTaskInstances) &&
+				partitionMappings.equals(that.partitionMappings);
+		}
+
+		public int[] getOldTaskInstances(int defaultSubtask) {
+			return oldTaskInstances.equals(NO_SUBTASKS) ?
+				new int[] {defaultSubtask} :
+				oldTaskInstances.stream().toArray();
+		}
+
+		public PartitionMapping getPartitionMapping(int partitionIndex) {
+			if (partitionMappings.isEmpty()) {
+				return NO_CHANNEL_MAPPING;
+			}
+			return partitionMappings.get(partitionIndex);
+		}
+
+		@Override
+		public int hashCode() {
+			return Objects.hash(oldTaskInstances, partitionMappings);
+		}
+
+		@Override
+		public String toString() {
+			return "VirtualChannelMapping{" +
+				"oldTaskInstances=" + oldTaskInstances +
+				", partitionMappings=" + partitionMappings +
+				'}';
+		}
+	}
+
+	/**
+	 * Contains the fine-grain channel mappings that occur when a connected operator has been rescaled.
+	 */
+	public static class PartitionMapping implements Serializable {
+
+		/**
+		 * For each new channel (=index), all old channels are set.
+		 */
+		private final List<BitSet> newToOldChannelIndexes;

Review comment:
       Again, the choice of data structures isn't obvious. Why not `Map<Int, Int>`?
   
   (ditto `oldToNewChannelIndexes`)

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/AbstractChannelStateHandle.java
##########
@@ -46,6 +46,11 @@
 	private final List<Long> offsets;
 	private final long size;
 
+	/**
+	 * The original subtask index before rescaling recovery.
+	 */
+	private int originalSubtaskIndex;

Review comment:
       You probably should serialize this field and add it to the equals/hashCode.

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -327,4 +289,252 @@ public boolean hasState() {
 			|| inputChannelState.hasState()
 			|| resultSubpartitionState.hasState();
 	}
+
+	public static Builder builder() {
+		return new Builder();
+	}
+
+	/**
+	 * The builder for a new {@link OperatorSubtaskState} which can be obtained by {@link #builder()}.
+	 */
+	public static class Builder {
+		private StateObjectCollection<OperatorStateHandle> managedOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<OperatorStateHandle> rawOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> managedKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> rawKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<InputChannelStateHandle> inputChannelState = StateObjectCollection.empty();
+		private StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState = StateObjectCollection.empty();
+		private VirtualChannelMapping inputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+		private VirtualChannelMapping outputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+
+		private Builder() {
+		}
+
+		public Builder setManagedOperatorState(StateObjectCollection<OperatorStateHandle> managedOperatorState) {
+			this.managedOperatorState = checkNotNull(managedOperatorState);
+			return this;
+		}
+
+		public Builder setManagedOperatorState(OperatorStateHandle managedOperatorState) {
+			return setManagedOperatorState(StateObjectCollection.singleton(checkNotNull(managedOperatorState)));
+		}
+
+		public Builder setRawOperatorState(StateObjectCollection<OperatorStateHandle> rawOperatorState) {
+			this.rawOperatorState = checkNotNull(rawOperatorState);
+			return this;
+		}
+
+		public Builder setRawOperatorState(OperatorStateHandle rawOperatorState) {
+			return setRawOperatorState(StateObjectCollection.singleton(checkNotNull(rawOperatorState)));
+		}
+
+		public Builder setManagedKeyedState(StateObjectCollection<KeyedStateHandle> managedKeyedState) {
+			this.managedKeyedState = checkNotNull(managedKeyedState);
+			return this;
+		}
+
+		public Builder setManagedKeyedState(KeyedStateHandle managedKeyedState) {
+			return setManagedKeyedState(StateObjectCollection.singleton(checkNotNull(managedKeyedState)));
+		}
+
+		public Builder setRawKeyedState(StateObjectCollection<KeyedStateHandle> rawKeyedState) {
+			this.rawKeyedState = checkNotNull(rawKeyedState);
+			return this;
+		}
+
+		public Builder setRawKeyedState(KeyedStateHandle rawKeyedState) {
+			return setRawKeyedState(StateObjectCollection.singleton(checkNotNull(rawKeyedState)));
+		}
+
+		public Builder setInputChannelState(StateObjectCollection<InputChannelStateHandle> inputChannelState) {
+			this.inputChannelState = checkNotNull(inputChannelState);
+			return this;
+		}
+
+		public Builder setResultSubpartitionState(StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState) {
+			this.resultSubpartitionState = checkNotNull(resultSubpartitionState);
+			return this;
+		}
+
+		public Builder setInputChannelMappings(VirtualChannelMapping inputChannelMappings) {
+			this.inputChannelMappings = checkNotNull(inputChannelMappings);
+			return this;
+		}
+
+		public Builder setOutputChannelMappings(VirtualChannelMapping outputChannelMappings) {
+			this.outputChannelMappings = checkNotNull(outputChannelMappings);
+			return this;
+		}
+
+		public OperatorSubtaskState build() {
+			return new OperatorSubtaskState(
+				managedOperatorState,
+				rawOperatorState,
+				managedKeyedState,
+				rawKeyedState,
+				inputChannelState,
+				resultSubpartitionState,
+				inputChannelMappings,
+				outputChannelMappings);
+		}
+	}
+
+	/**
+	 * Captures ambiguous mappings of old channels to new channels.
+	 *
+	 * <p>For inputs, this mapping implies the following:
+	 * <li>
+	 *     <ul>{@link #oldTaskInstances} is set when there is a rescale on this task potentially leading to different
+	 *     key groups. Upstream task has a corresponding {@link #partitionMappings} where it sends data over
+	 *     virtual channel while specifying the channel index in the VirtualChannelSelector. This subtask then
+	 *     demultiplexes over the virtual subtask index.</ul>
+	 *     <ul>{@link #partitionMappings} is set when there is a downscale of the upstream task. Upstream task has
+	 *     a corresponding {@link #oldTaskInstances} where it sends data over virtual channel while specifying the
+	 *     subtask index in the VirtualChannelSelector. This subtask then demultiplexes over channel indexes.</ul>
+	 * </li>
+	 *
+	 * <p>For outputs, it's vice-versa. The information must be kept in sync but they are used in opposite ways for
+	 * multiplexing/demultiplexing.
+	 *
+	 * <p>Note that in the common rescaling case both information is set and need to be simultaneously used. If the
+	 * input subtask subsumes the state of 3 old subtasks and a channel corresponds to 2 old channels, then there are
+	 * 6 virtual channels to be demultiplexed.
+	 */
+	public static class VirtualChannelMapping implements Serializable {
+		public static final PartitionMapping NO_CHANNEL_MAPPING = new PartitionMapping(emptyList());
+		public static final List<PartitionMapping> NO_PARTITIONS = emptyList();
+		public static final BitSet NO_SUBTASKS = new BitSet();
+		public static final VirtualChannelMapping NO_MAPPING = new VirtualChannelMapping(NO_SUBTASKS, NO_PARTITIONS);
+
+		/**
+		 * Set when several operator instances are merged into one.
+		 */
+		private final BitSet oldTaskInstances;
+
+		/**
+		 * Set when channels are merged because the connected operator has been rescaled.
+		 */
+		private final List<PartitionMapping> partitionMappings;
+
+		public VirtualChannelMapping(BitSet oldTaskInstances, List<PartitionMapping> partitionMappings) {
+			this.oldTaskInstances = oldTaskInstances;
+			this.partitionMappings = partitionMappings;
+		}
+
+		@Override
+		public boolean equals(Object o) {
+			if (this == o) {
+				return true;
+			}
+			if (o == null || getClass() != o.getClass()) {
+				return false;
+			}
+			final VirtualChannelMapping that = (VirtualChannelMapping) o;
+			return oldTaskInstances.equals(that.oldTaskInstances) &&
+				partitionMappings.equals(that.partitionMappings);
+		}
+
+		public int[] getOldTaskInstances(int defaultSubtask) {
+			return oldTaskInstances.equals(NO_SUBTASKS) ?
+				new int[] {defaultSubtask} :
+				oldTaskInstances.stream().toArray();
+		}
+
+		public PartitionMapping getPartitionMapping(int partitionIndex) {
+			if (partitionMappings.isEmpty()) {
+				return NO_CHANNEL_MAPPING;
+			}
+			return partitionMappings.get(partitionIndex);
+		}
+
+		@Override
+		public int hashCode() {
+			return Objects.hash(oldTaskInstances, partitionMappings);
+		}
+
+		@Override
+		public String toString() {
+			return "VirtualChannelMapping{" +
+				"oldTaskInstances=" + oldTaskInstances +
+				", partitionMappings=" + partitionMappings +
+				'}';
+		}
+	}
+
+	/**
+	 * Contains the fine-grain channel mappings that occur when a connected operator has been rescaled.
+	 */
+	public static class PartitionMapping implements Serializable {

Review comment:
       1. `PartitionMapping` associates with the `ResultPartition`, but this class maps input channels and subPartitions, right? How about `RescaledChannelsMapping`?
   2. `serialVersionUID`?
   3. nit: I'd extract this class too

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
##########
@@ -620,18 +758,112 @@ private static void checkStateMappingCompleteness(
 			chainOpParallelStates,
 			oldParallelism,
 			newParallelism);
+	}
+
+	static class TaskStateAssignment {
+		final ExecutionJobVertex executionJobVertex;
+		final Map<OperatorID, OperatorState> oldState;
+		final boolean hasState;
+		final int newParallelism;
+		final OperatorID inputOperatorID;
+		final OperatorID outputOperatorID;
+
+		final Map<OperatorInstanceID, List<OperatorStateHandle>> subManagedOperatorState;
+		final Map<OperatorInstanceID, List<OperatorStateHandle>> subRawOperatorState;
+		final Map<OperatorInstanceID, List<KeyedStateHandle>> subManagedKeyedState;
+		final Map<OperatorInstanceID, List<KeyedStateHandle>> subRawKeyedState;
+
+		final Map<OperatorInstanceID, List<InputChannelStateHandle>> inputChannelStates;
+		final Map<OperatorInstanceID, List<ResultSubpartitionStateHandle>> resultSubpartitionStates;
+		/** The subpartitions mappings per partition set when the output operator for a partition was rescaled. */
+		List<BitSet> outputOperatorInstanceMappings = emptyList();
+		/** The input channel mappings per input set when the input operator for a gate was rescaled. */
+		List<BitSet> inputOperatorInstanceMappings = emptyList();
+		/** The subpartitions mappings of the upstream task per input set when its output operator was rescaled. */
+		final Map<Integer, List<BitSet>> upstreamVirtualChannels;
+		/** The input channel mappings of the downstream task per partition set when its input operator was rescaled. */
+		final Map<Integer, List<BitSet>> downStreamVirtualChannels;

Review comment:
       How about storing a reference to the downstream/upstream `TaskStateAssignment` and get the mappings from there?
   I think it would be more readable.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518163988



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/ChannelRescalerRepartitioner.java
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.checkpoint;
+
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.io.network.api.writer.ChannelStateRescaler;
+
+import org.apache.flink.shaded.guava18.com.google.common.collect.Iterables;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * A repartitioner that assigns the same channel state to multiple subtasks according to some mapping.
+ *
+ * <p>The replicated data will then be filtered before processing the record.
+ *
+ * <p>Note that channel mappings are cached for the same parallelism changes.
+ */
+public class ChannelRescalerRepartitioner<T> implements OperatorStateRepartitioner<T> {
+	private final ChannelStateRescaler channelStateRescaler;
+	private final Map<Tuple2<Integer, Integer>, Map<Integer, Set<Integer>>> newToOldMappingCache = new HashMap<>(2);
+
+	public ChannelRescalerRepartitioner(ChannelStateRescaler channelStateRescaler) {
+		this.channelStateRescaler = channelStateRescaler;
+	}
+
+	private static <T> List<T> getOldState(List<List<T>> previousParallelSubtaskStates, Set<Integer> oldIndexes) {
+		switch (oldIndexes.size()) {
+			case 0:
+				return Collections.emptyList();
+			case 1:
+				return previousParallelSubtaskStates.get(Iterables.getOnlyElement(oldIndexes));
+			default:
+				return oldIndexes.stream()
+					.flatMap(oldIndex -> previousParallelSubtaskStates.get(oldIndex).stream())
+					.collect(Collectors.toList());
+		}
+	}
+
+	protected Map<Integer, Set<Integer>> createNewToOldMapping(int oldParallelism, int newParallelism) {
+		return IntStream.range(0, newParallelism).boxed().
+			collect(Collectors.toMap(
+				Function.identity(),
+				channelIndex -> channelStateRescaler.getOldChannels(
+					channelIndex,
+					oldParallelism,
+					newParallelism)));
+	}
+
+	@Override
+	public List<List<T>> repartitionState(
+			List<List<T>> previousParallelSubtaskStates,
+			int oldParallelism,
+			int newParallelism) {
+		final Map<Integer, Set<Integer>> newToOldMapping = getNewToOldMapping(oldParallelism, newParallelism);
+		return IntStream.range(0, newParallelism)
+			.mapToObj(newIndex -> getOldState(previousParallelSubtaskStates, newToOldMapping.get(newIndex)))
+			.collect(Collectors.toList());

Review comment:
       It's good feedback if stuff is confusing. I'll offer an alternative that avoids using streams:
   
   ```
   		List<List<T>> repartitioned = new ArrayList<>();
   		for (int newIndex = 0; newIndex < newParallelism; newIndex++) {
   			repartitioned.add(getOldState(previousParallelSubtaskStates, newToOldSubtasksMapping.get(newIndex)));
   		}
   		return repartitioned;
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517334175



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -327,4 +289,252 @@ public boolean hasState() {
 			|| inputChannelState.hasState()
 			|| resultSubpartitionState.hasState();
 	}
+
+	public static Builder builder() {
+		return new Builder();
+	}
+
+	/**
+	 * The builder for a new {@link OperatorSubtaskState} which can be obtained by {@link #builder()}.
+	 */
+	public static class Builder {
+		private StateObjectCollection<OperatorStateHandle> managedOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<OperatorStateHandle> rawOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> managedKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> rawKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<InputChannelStateHandle> inputChannelState = StateObjectCollection.empty();
+		private StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState = StateObjectCollection.empty();
+		private VirtualChannelMapping inputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+		private VirtualChannelMapping outputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+
+		private Builder() {
+		}
+
+		public Builder setManagedOperatorState(StateObjectCollection<OperatorStateHandle> managedOperatorState) {
+			this.managedOperatorState = checkNotNull(managedOperatorState);
+			return this;
+		}
+
+		public Builder setManagedOperatorState(OperatorStateHandle managedOperatorState) {
+			return setManagedOperatorState(StateObjectCollection.singleton(checkNotNull(managedOperatorState)));
+		}
+
+		public Builder setRawOperatorState(StateObjectCollection<OperatorStateHandle> rawOperatorState) {
+			this.rawOperatorState = checkNotNull(rawOperatorState);
+			return this;
+		}
+
+		public Builder setRawOperatorState(OperatorStateHandle rawOperatorState) {
+			return setRawOperatorState(StateObjectCollection.singleton(checkNotNull(rawOperatorState)));
+		}
+
+		public Builder setManagedKeyedState(StateObjectCollection<KeyedStateHandle> managedKeyedState) {
+			this.managedKeyedState = checkNotNull(managedKeyedState);
+			return this;
+		}
+
+		public Builder setManagedKeyedState(KeyedStateHandle managedKeyedState) {
+			return setManagedKeyedState(StateObjectCollection.singleton(checkNotNull(managedKeyedState)));
+		}
+
+		public Builder setRawKeyedState(StateObjectCollection<KeyedStateHandle> rawKeyedState) {
+			this.rawKeyedState = checkNotNull(rawKeyedState);
+			return this;
+		}
+
+		public Builder setRawKeyedState(KeyedStateHandle rawKeyedState) {
+			return setRawKeyedState(StateObjectCollection.singleton(checkNotNull(rawKeyedState)));
+		}
+
+		public Builder setInputChannelState(StateObjectCollection<InputChannelStateHandle> inputChannelState) {
+			this.inputChannelState = checkNotNull(inputChannelState);
+			return this;
+		}
+
+		public Builder setResultSubpartitionState(StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState) {
+			this.resultSubpartitionState = checkNotNull(resultSubpartitionState);
+			return this;
+		}
+
+		public Builder setInputChannelMappings(VirtualChannelMapping inputChannelMappings) {
+			this.inputChannelMappings = checkNotNull(inputChannelMappings);
+			return this;
+		}
+
+		public Builder setOutputChannelMappings(VirtualChannelMapping outputChannelMappings) {
+			this.outputChannelMappings = checkNotNull(outputChannelMappings);
+			return this;
+		}
+
+		public OperatorSubtaskState build() {
+			return new OperatorSubtaskState(
+				managedOperatorState,
+				rawOperatorState,
+				managedKeyedState,
+				rawKeyedState,
+				inputChannelState,
+				resultSubpartitionState,
+				inputChannelMappings,
+				outputChannelMappings);
+		}
+	}
+
+	/**
+	 * Captures ambiguous mappings of old channels to new channels.
+	 *
+	 * <p>For inputs, this mapping implies the following:
+	 * <li>
+	 *     <ul>{@link #oldTaskInstances} is set when there is a rescale on this task potentially leading to different
+	 *     key groups. Upstream task has a corresponding {@link #partitionMappings} where it sends data over
+	 *     virtual channel while specifying the channel index in the VirtualChannelSelector. This subtask then
+	 *     demultiplexes over the virtual subtask index.</ul>
+	 *     <ul>{@link #partitionMappings} is set when there is a downscale of the upstream task. Upstream task has
+	 *     a corresponding {@link #oldTaskInstances} where it sends data over virtual channel while specifying the
+	 *     subtask index in the VirtualChannelSelector. This subtask then demultiplexes over channel indexes.</ul>
+	 * </li>
+	 *
+	 * <p>For outputs, it's vice-versa. The information must be kept in sync but they are used in opposite ways for
+	 * multiplexing/demultiplexing.
+	 *
+	 * <p>Note that in the common rescaling case both information is set and need to be simultaneously used. If the
+	 * input subtask subsumes the state of 3 old subtasks and a channel corresponds to 2 old channels, then there are
+	 * 6 virtual channels to be demultiplexed.
+	 */
+	public static class VirtualChannelMapping implements Serializable {
+		public static final PartitionMapping NO_CHANNEL_MAPPING = new PartitionMapping(emptyList());
+		public static final List<PartitionMapping> NO_PARTITIONS = emptyList();
+		public static final BitSet NO_SUBTASKS = new BitSet();
+		public static final VirtualChannelMapping NO_MAPPING = new VirtualChannelMapping(NO_SUBTASKS, NO_PARTITIONS);
+
+		/**
+		 * Set when several operator instances are merged into one.
+		 */
+		private final BitSet oldTaskInstances;
+
+		/**
+		 * Set when channels are merged because the connected operator has been rescaled.
+		 */
+		private final List<PartitionMapping> partitionMappings;

Review comment:
       Isn't array and array-list the same in complexity? I usually use a map for sparse data and list when it's dense (ideally consecutive), since it's much better on the memory consumption.
   But I could go for array. At some point, I didn't have `PartitionMapping` and used generics which don't mix well with arrays.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 5e5340241b5811243463f96dda7455b2869cf268 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223) 
   * cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517334175



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -327,4 +289,252 @@ public boolean hasState() {
 			|| inputChannelState.hasState()
 			|| resultSubpartitionState.hasState();
 	}
+
+	public static Builder builder() {
+		return new Builder();
+	}
+
+	/**
+	 * The builder for a new {@link OperatorSubtaskState} which can be obtained by {@link #builder()}.
+	 */
+	public static class Builder {
+		private StateObjectCollection<OperatorStateHandle> managedOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<OperatorStateHandle> rawOperatorState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> managedKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<KeyedStateHandle> rawKeyedState = StateObjectCollection.empty();
+		private StateObjectCollection<InputChannelStateHandle> inputChannelState = StateObjectCollection.empty();
+		private StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState = StateObjectCollection.empty();
+		private VirtualChannelMapping inputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+		private VirtualChannelMapping outputChannelMappings = VirtualChannelMapping.NO_MAPPING;
+
+		private Builder() {
+		}
+
+		public Builder setManagedOperatorState(StateObjectCollection<OperatorStateHandle> managedOperatorState) {
+			this.managedOperatorState = checkNotNull(managedOperatorState);
+			return this;
+		}
+
+		public Builder setManagedOperatorState(OperatorStateHandle managedOperatorState) {
+			return setManagedOperatorState(StateObjectCollection.singleton(checkNotNull(managedOperatorState)));
+		}
+
+		public Builder setRawOperatorState(StateObjectCollection<OperatorStateHandle> rawOperatorState) {
+			this.rawOperatorState = checkNotNull(rawOperatorState);
+			return this;
+		}
+
+		public Builder setRawOperatorState(OperatorStateHandle rawOperatorState) {
+			return setRawOperatorState(StateObjectCollection.singleton(checkNotNull(rawOperatorState)));
+		}
+
+		public Builder setManagedKeyedState(StateObjectCollection<KeyedStateHandle> managedKeyedState) {
+			this.managedKeyedState = checkNotNull(managedKeyedState);
+			return this;
+		}
+
+		public Builder setManagedKeyedState(KeyedStateHandle managedKeyedState) {
+			return setManagedKeyedState(StateObjectCollection.singleton(checkNotNull(managedKeyedState)));
+		}
+
+		public Builder setRawKeyedState(StateObjectCollection<KeyedStateHandle> rawKeyedState) {
+			this.rawKeyedState = checkNotNull(rawKeyedState);
+			return this;
+		}
+
+		public Builder setRawKeyedState(KeyedStateHandle rawKeyedState) {
+			return setRawKeyedState(StateObjectCollection.singleton(checkNotNull(rawKeyedState)));
+		}
+
+		public Builder setInputChannelState(StateObjectCollection<InputChannelStateHandle> inputChannelState) {
+			this.inputChannelState = checkNotNull(inputChannelState);
+			return this;
+		}
+
+		public Builder setResultSubpartitionState(StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState) {
+			this.resultSubpartitionState = checkNotNull(resultSubpartitionState);
+			return this;
+		}
+
+		public Builder setInputChannelMappings(VirtualChannelMapping inputChannelMappings) {
+			this.inputChannelMappings = checkNotNull(inputChannelMappings);
+			return this;
+		}
+
+		public Builder setOutputChannelMappings(VirtualChannelMapping outputChannelMappings) {
+			this.outputChannelMappings = checkNotNull(outputChannelMappings);
+			return this;
+		}
+
+		public OperatorSubtaskState build() {
+			return new OperatorSubtaskState(
+				managedOperatorState,
+				rawOperatorState,
+				managedKeyedState,
+				rawKeyedState,
+				inputChannelState,
+				resultSubpartitionState,
+				inputChannelMappings,
+				outputChannelMappings);
+		}
+	}
+
+	/**
+	 * Captures ambiguous mappings of old channels to new channels.
+	 *
+	 * <p>For inputs, this mapping implies the following:
+	 * <li>
+	 *     <ul>{@link #oldTaskInstances} is set when there is a rescale on this task potentially leading to different
+	 *     key groups. Upstream task has a corresponding {@link #partitionMappings} where it sends data over
+	 *     virtual channel while specifying the channel index in the VirtualChannelSelector. This subtask then
+	 *     demultiplexes over the virtual subtask index.</ul>
+	 *     <ul>{@link #partitionMappings} is set when there is a downscale of the upstream task. Upstream task has
+	 *     a corresponding {@link #oldTaskInstances} where it sends data over virtual channel while specifying the
+	 *     subtask index in the VirtualChannelSelector. This subtask then demultiplexes over channel indexes.</ul>
+	 * </li>
+	 *
+	 * <p>For outputs, it's vice-versa. The information must be kept in sync but they are used in opposite ways for
+	 * multiplexing/demultiplexing.
+	 *
+	 * <p>Note that in the common rescaling case both information is set and need to be simultaneously used. If the
+	 * input subtask subsumes the state of 3 old subtasks and a channel corresponds to 2 old channels, then there are
+	 * 6 virtual channels to be demultiplexed.
+	 */
+	public static class VirtualChannelMapping implements Serializable {
+		public static final PartitionMapping NO_CHANNEL_MAPPING = new PartitionMapping(emptyList());
+		public static final List<PartitionMapping> NO_PARTITIONS = emptyList();
+		public static final BitSet NO_SUBTASKS = new BitSet();
+		public static final VirtualChannelMapping NO_MAPPING = new VirtualChannelMapping(NO_SUBTASKS, NO_PARTITIONS);
+
+		/**
+		 * Set when several operator instances are merged into one.
+		 */
+		private final BitSet oldTaskInstances;
+
+		/**
+		 * Set when channels are merged because the connected operator has been rescaled.
+		 */
+		private final List<PartitionMapping> partitionMappings;

Review comment:
       Isn't array and array-list the same in complexity? I usually use a map for sparse data and list when it's dense (ideally consecutive), since it's much better on the memory consumption.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-722914106


   > Thanks for updating the PR, LGTM.
   > 
   > I'm only concerned about changing the partitioner upon restart. For example, if the job is modified so that DOP of upstream is no longer equal to DOP of the downstream; then it switches from `Forward` to `Rebalance` IIUC. Which will break state distribution, so it should be prohibited.
   > If that's a valid concern I'd approach it in a different effort.
   
   The concern is valid. I'm adding a check in the second PR then.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518555888



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -64,81 +61,64 @@
 	/**
 	 * Snapshot from the {@link org.apache.flink.runtime.state.OperatorStateBackend}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<OperatorStateHandle> managedOperatorState;
 
 	/**
 	 * Snapshot written using {@link org.apache.flink.runtime.state.OperatorStateCheckpointOutputStream}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<OperatorStateHandle> rawOperatorState;
 
 	/**
 	 * Snapshot from {@link org.apache.flink.runtime.state.KeyedStateBackend}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<KeyedStateHandle> managedKeyedState;
 
 	/**
 	 * Snapshot written using {@link org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<KeyedStateHandle> rawKeyedState;
 
-	@Nonnull
 	private final StateObjectCollection<InputChannelStateHandle> inputChannelState;
 
-	@Nonnull
 	private final StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState;
 
 	/**
-	 * The state size. This is also part of the deserialized state handle.
-	 * We store it here in order to not deserialize the state handle when
-	 * gathering stats.
+	 * The subpartitions mappings per partition set when the output operator for a partition was rescaled. The key is
+	 * the partition id and the value contains all subtask indexes of the output operator before rescaling.
 	 */
-	private final long stateSize;
+	private final InflightDataRescalingDescriptor inputRescalingDescriptor;

Review comment:
       Added a comment.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517340169



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/AbstractChannelStateHandle.java
##########
@@ -46,6 +46,11 @@
 	private final List<Long> offsets;
 	private final long size;
 
+	/**
+	 * The original subtask index before rescaling recovery.
+	 */
+	private int originalSubtaskIndex;

Review comment:
       It is serialized when transferring the `OperatorSubtaskState` through `Serializable`. It's currently not needed in the checkpoints (as the information can be easily inferred). Is it still necessary from your perspective?
   I'd add it to `equals/hashCode` anyways.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518428698



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/ForwardPartitioner.java
##########
@@ -43,4 +44,9 @@ public int selectChannel(SerializationDelegate<StreamRecord<T>> record) {
 	public String toString() {
 		return "FORWARD";
 	}
+
+	@Override
+	public ChannelStateRescaler getDownstreamChannelStateRescaler() {
+		return ChannelStateRescaler.FIRST_CHANNEL;

Review comment:
       Yes, `ROUND_ROBIN` makes sense here and the new naming looks good!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518033137



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/CustomPartitionerWrapper.java
##########
@@ -55,6 +56,14 @@ public int selectChannel(SerializationDelegate<StreamRecord<T>> record) {
 		return partitioner.partition(key, numberOfChannels);
 	}
 
+	@Override
+	public ChannelStateRescaler getDownstreamChannelStateRescaler() {
+		// fully rely on filtering downstream
+		// note that custom partitioners are not officially supported - the user has to force rescaling
+		// in that case, we assume that the custom partitioner is deterministic
+		return ChannelStateRescaler.BROADCAST;

Review comment:
       `BROADCAST` would replicate each channel state to each subtask (very costly) including all new states. All superfluous replicas are filtered out downstream by the partitioner.
   Note that I will extend your check for cyclic graphs to customer partitioners for disabling UC. So the user has to force it to make it work. However, I do not see a reason to then fail on the second level again.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517176106



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java
##########
@@ -692,6 +692,8 @@ private void connect(Integer headOfChain, StreamEdge edge) {
 		}
 		// set strategy name so that web interface can show it.
 		jobEdge.setShipStrategyName(partitioner.toString());
+		jobEdge.setDownstreamChannelStateRescaler(partitioner.getUpstreamChannelStateRescaler());
+		jobEdge.setUpstreamChannelStateRescaler(partitioner.getDownstreamChannelStateRescaler());

Review comment:
       There is no good reason except me screwing up. I did a couple of renamings prior to republishing PR and apparently renamed things asymetrically. I adjusted it (so upstream is now set to upstream).
   Sorry for the confusion and good catch.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] pnowojski commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r512158143



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/ChannelRescaler.java
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.api.writer;
+
+import org.apache.flink.annotation.Internal;
+
+import java.io.Serializable;
+import java.util.BitSet;
+
+/**
+ * The {@link ChannelRescaler} narrows down the channels that need to be read during rescaling to recover from a
+ * particular channel when in-flight data has been stored in the checkpoint.
+ */
+@Internal
+public interface ChannelRescaler extends Serializable {
+	/**
+	 * Returns all old channel indexes that need to be read to restore all buffers for the given new channel index on
+	 * rescale.
+	 */
+	BitSet rescaleIntersections(int newChannelIndex, int oldNumberOfChannels, int newNumberOfChannels);

Review comment:
       What is the relation between this method/interface and `#rescaleIntersections` from the `Partitioner`? Why one returns ` int[]` and the other `BitSet`?

##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -63,31 +61,25 @@
 	/**
 	 * Snapshot from the {@link org.apache.flink.runtime.state.OperatorStateBackend}.
 	 */
-	@Nonnull

Review comment:
       Removal of `@Nonnull` is controversial by some. Some people prefer it as cheap, debug mode only assertion. This is because as far as I know, `@Nonnull` in debug mode, is adding `checkNonNull(...)` equivalents.
   
   I personally would be fine by not using them, and I personally I'm not adding them in a new code, but because of the above reason, I would be actually against removing them from a code that someone added (otherwise, we can end up in a ping pong situation when you are removing them and someone else re-adding).

##########
File path: flink-core/src/main/java/org/apache/flink/api/common/functions/Partitioner.java
##########
@@ -37,4 +40,20 @@
 	 * @return The partition index.
 	 */
 	int partition(K key, int numPartitions);
+
+	/**
+	 * Returns all partitions that need to be read to restore the given new partition. The partitioner is then
+	 * applied on the key of the restored record to filter all irrelevant records.
+	 *
+	 * <p>In particular, to create a partition X after rescaling, all partitions returned by this method are fully read
+	 * and the key of each record is then fed into {@link #partition(Object, int)} to check if it belongs to X.
+	 *
+	 * <p>The default implementation states that all partitions need to be scanned and should be overwritten to improve
+	 * performance.
+	 */
+	@PublicEvolving
+	default int[] rescaleIntersections(int newPartition, int oldNumPartitions, int newNumPartitions) {
+		// any old partition may contain a record that should be in the new partition after rescaling
+		return IntStream.range(0, oldNumPartitions).toArray();
+	}

Review comment:
       We need this to support `CustomPartitionerWrapper`, right? This is `@Public` interface :( I think we need to think twice before we commit ourselves to such change. Let's sync offline again whether this is really the best/the only way of solving our problem.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 7c42ceb0f3c54b0ab63f053ce91a20c4866119d1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172) 
   * 1b1a34de33af36c301266d175cb72151b1bc51c4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116) 
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ee73393a13c999be626f7b8b98a170be88a093d9 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228",
       "triggerID" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8233",
       "triggerID" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 88a90896e0cf1c7bf27fe1aa40158c7280c5fa58 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228) 
   * 25b0c065fae8c72fa607c5b6f47376ff3e6f8568 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8233) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 6bfcf5914a2742ce41f05b2b77b70f7df85fc696 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198) 
   * 470d466cfe4455aefd5a302cedd92b57750f2c57 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 7c42ceb0f3c54b0ab63f053ce91a20c4866119d1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172) 
   * 1b1a34de33af36c301266d175cb72151b1bc51c4 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228",
       "triggerID" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224) 
   * 88a90896e0cf1c7bf27fe1aa40158c7280c5fa58 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228) 
   * 25b0c065fae8c72fa607c5b6f47376ff3e6f8568 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517337552



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
##########
@@ -99,32 +119,40 @@ public void assignStates() {
 						operatorID,
 						executionJobVertex.getParallelism(),
 						executionJobVertex.getMaxParallelism());
-				} else if (operatorState.getNumberCollectedStates() > 0) {
-					statelessSubTasks = false;
 				}
-				operatorStates.add(operatorState);
+				operatorStates.put(operatorIDPair.getGeneratedOperatorID(), operatorState);
 			}
-			if (!statelessSubTasks) { // skip tasks where no operator has any state
-				assignAttemptState(executionJobVertex, operatorStates);
+
+			final TaskStateAssignment stateAssignment = new TaskStateAssignment(executionJobVertex,	operatorStates);
+			vertexAssignments.put(executionJobVertex, stateAssignment);
+			for (final IntermediateResult producedDataSet : executionJobVertex.getInputs()) {
+				consumerAssignment.put(producedDataSet, stateAssignment);

Review comment:
       `UnionGate` reads two or more different `IntermediateResult`. `IntermediateResult` translates directly into subpartition/gate.
   However, it's a good point that `IntermediateResult` doesn't override equals/hashCode. It currently works, because the JobGraph is a graph where the objects are connected, but I don't know if we can rely on the assumption. I'm switching to `IntermediateDataSetID`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518555888



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -64,81 +61,64 @@
 	/**
 	 * Snapshot from the {@link org.apache.flink.runtime.state.OperatorStateBackend}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<OperatorStateHandle> managedOperatorState;
 
 	/**
 	 * Snapshot written using {@link org.apache.flink.runtime.state.OperatorStateCheckpointOutputStream}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<OperatorStateHandle> rawOperatorState;
 
 	/**
 	 * Snapshot from {@link org.apache.flink.runtime.state.KeyedStateBackend}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<KeyedStateHandle> managedKeyedState;
 
 	/**
 	 * Snapshot written using {@link org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<KeyedStateHandle> rawKeyedState;
 
-	@Nonnull
 	private final StateObjectCollection<InputChannelStateHandle> inputChannelState;
 
-	@Nonnull
 	private final StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState;
 
 	/**
-	 * The state size. This is also part of the deserialized state handle.
-	 * We store it here in order to not deserialize the state handle when
-	 * gathering stats.
+	 * The subpartitions mappings per partition set when the output operator for a partition was rescaled. The key is
+	 * the partition id and the value contains all subtask indexes of the output operator before rescaling.
 	 */
-	private final long stateSize;
+	private final InflightDataRescalingDescriptor inputRescalingDescriptor;

Review comment:
       Added a comment.
   
   ```
   Note that this field is only set by {@link StateAssignmentOperation} and will not be persisted in the checkpoint itself as it can only be calculated if the the post-recovery scale factor is known.
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r517203045



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/StateAssignmentOperation.java
##########
@@ -620,18 +758,112 @@ private static void checkStateMappingCompleteness(
 			chainOpParallelStates,
 			oldParallelism,
 			newParallelism);
+	}
+
+	static class TaskStateAssignment {
+		final ExecutionJobVertex executionJobVertex;
+		final Map<OperatorID, OperatorState> oldState;
+		final boolean hasState;
+		final int newParallelism;
+		final OperatorID inputOperatorID;
+		final OperatorID outputOperatorID;
+
+		final Map<OperatorInstanceID, List<OperatorStateHandle>> subManagedOperatorState;
+		final Map<OperatorInstanceID, List<OperatorStateHandle>> subRawOperatorState;
+		final Map<OperatorInstanceID, List<KeyedStateHandle>> subManagedKeyedState;
+		final Map<OperatorInstanceID, List<KeyedStateHandle>> subRawKeyedState;
+
+		final Map<OperatorInstanceID, List<InputChannelStateHandle>> inputChannelStates;
+		final Map<OperatorInstanceID, List<ResultSubpartitionStateHandle>> resultSubpartitionStates;
+		/** The subpartitions mappings per partition set when the output operator for a partition was rescaled. */
+		List<BitSet> outputOperatorInstanceMappings = emptyList();
+		/** The input channel mappings per input set when the input operator for a gate was rescaled. */
+		List<BitSet> inputOperatorInstanceMappings = emptyList();
+		/** The subpartitions mappings of the upstream task per input set when its output operator was rescaled. */
+		final Map<Integer, List<BitSet>> upstreamVirtualChannels;
+		/** The input channel mappings of the downstream task per partition set when its input operator was rescaled. */
+		final Map<Integer, List<BitSet>> downStreamVirtualChannels;

Review comment:
       I like it!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] rkhachatryan commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
rkhachatryan commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518425981



##########
File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/OperatorSubtaskState.java
##########
@@ -64,81 +61,64 @@
 	/**
 	 * Snapshot from the {@link org.apache.flink.runtime.state.OperatorStateBackend}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<OperatorStateHandle> managedOperatorState;
 
 	/**
 	 * Snapshot written using {@link org.apache.flink.runtime.state.OperatorStateCheckpointOutputStream}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<OperatorStateHandle> rawOperatorState;
 
 	/**
 	 * Snapshot from {@link org.apache.flink.runtime.state.KeyedStateBackend}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<KeyedStateHandle> managedKeyedState;
 
 	/**
 	 * Snapshot written using {@link org.apache.flink.runtime.state.KeyedStateCheckpointOutputStream}.
 	 */
-	@Nonnull
 	private final StateObjectCollection<KeyedStateHandle> rawKeyedState;
 
-	@Nonnull
 	private final StateObjectCollection<InputChannelStateHandle> inputChannelState;
 
-	@Nonnull
 	private final StateObjectCollection<ResultSubpartitionStateHandle> resultSubpartitionState;
 
 	/**
-	 * The state size. This is also part of the deserialized state handle.
-	 * We store it here in order to not deserialize the state handle when
-	 * gathering stats.
+	 * The subpartitions mappings per partition set when the output operator for a partition was rescaled. The key is
+	 * the partition id and the value contains all subtask indexes of the output operator before rescaling.
 	 */
-	private final long stateSize;
+	private final InflightDataRescalingDescriptor inputRescalingDescriptor;

Review comment:
       Yes, you are right (nit: maybe comment that it's only computed upon assignment?)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on a change in pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518175408



##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/ForwardPartitioner.java
##########
@@ -43,4 +44,9 @@ public int selectChannel(SerializationDelegate<StreamRecord<T>> record) {
 	public String toString() {
 		return "FORWARD";
 	}
+
+	@Override
+	public ChannelStateRescaler getDownstreamChannelStateRescaler() {
+		return ChannelStateRescaler.FIRST_CHANNEL;

Review comment:
       Really good catch. I was confused by the selection process of ForwardPartitioner always selecting 0. But what we want is to have a round robin distribution (so assign same as before in case of upscaling), redistribute on downscale to avoid data loss.
   I used this find to rethink the nomenclature and actually everything that is related to `ChannelStateRescaler` is actually about subtask instance mapping, so I renamed to `SubtaskStateMapper`. Then it becomes immediately apparent that choosing the first task is plain wrong here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * 470d466cfe4455aefd5a302cedd92b57750f2c57 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212) 
   * 5e5340241b5811243463f96dda7455b2869cf268 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e081fecba324a0e35017f4b89cc0499c9112a768 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049) 
   * ee73393a13c999be626f7b8b98a170be88a093d9 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-714048414


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8049",
       "triggerID" : "e081fecba324a0e35017f4b89cc0499c9112a768",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8077",
       "triggerID" : "ee73393a13c999be626f7b8b98a170be88a093d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8116",
       "triggerID" : "c20d917f5ac0c8c3d932d0c0ba1ca1c7535c0b47",
       "triggerType" : "PUSH"
     }, {
       "hash" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "458aee4e1d6779beb640f563c7c9b5a65e645237",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8172",
       "triggerID" : "7c42ceb0f3c54b0ab63f053ce91a20c4866119d1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8179",
       "triggerID" : "1b1a34de33af36c301266d175cb72151b1bc51c4",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8198",
       "triggerID" : "6bfcf5914a2742ce41f05b2b77b70f7df85fc696",
       "triggerType" : "PUSH"
     }, {
       "hash" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8212",
       "triggerID" : "470d466cfe4455aefd5a302cedd92b57750f2c57",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5e5340241b5811243463f96dda7455b2869cf268",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8223",
       "triggerID" : "5e5340241b5811243463f96dda7455b2869cf268",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8224",
       "triggerID" : "cbc3b0ca32cfbfbf57ab34f85ae12e4f8c0ee4aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8228",
       "triggerID" : "88a90896e0cf1c7bf27fe1aa40158c7280c5fa58",
       "triggerType" : "PUSH"
     }, {
       "hash" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8233",
       "triggerID" : "25b0c065fae8c72fa607c5b6f47376ff3e6f8568",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8885",
       "triggerID" : "9cbbcb24751268c0b5289b86606eb28baa1287eb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bfd138a5fda6a4eeb4544cc88f9e5660063ab084",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9025",
       "triggerID" : "bfd138a5fda6a4eeb4544cc88f9e5660063ab084",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c95db355becc1079be6287aa7d165e7590f9fcf6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9104",
       "triggerID" : "722547278",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "c95db355becc1079be6287aa7d165e7590f9fcf6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9104",
       "triggerID" : "c95db355becc1079be6287aa7d165e7590f9fcf6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "722547278",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 458aee4e1d6779beb640f563c7c9b5a65e645237 UNKNOWN
   * c95db355becc1079be6287aa7d165e7590f9fcf6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9104) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] AHeise commented on pull request #13735: [FLINK-19533][checkpoint] Add channel state reassignment for unaligned checkpoints.

Posted by GitBox <gi...@apache.org>.
AHeise commented on pull request #13735:
URL: https://github.com/apache/flink/pull/13735#issuecomment-721344300


   As discussed offline, `ChannelRescaler` is now a simple enum to avoid any user code involvement into the `JobGraph`. Also in the current version, there are no changes to Public API anymore.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org