[jira] [Commented] (FLINK-5041) Implement savepoint backwards compatibility 1.1 -> 1.2

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-5041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15725646#comment-15725646 ] 

ASF GitHub Bot commented on FLINK-5041:
---------------------------------------

Github user StephanEwen commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2781#discussion_r91087211
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapKeyedStateBackend.java ---
    @@ -346,4 +359,130 @@ public String toString() {
     		return "HeapKeyedStateBackend";
     	}
     
    +	/**
    +	 * REMOVE
    +	 */
    +	@Internal
    +	@Deprecated
    +	public Map<String, StateTable<K, ?, ?>> getStateTables() {
    +		return stateTables;
    +	}
    +
    +	@Deprecated
    +	private void restoreOldSavepointKeyedState(
    +			Collection<KeyGroupsStateHandle> stateHandles) throws IOException, ClassNotFoundException {
    +
    +		if (stateHandles.isEmpty()) {
    +			return;
    +		}
    +
    +		HashMap<String, KvStateSnapshot<K, ?, ?, ?>> namedStates =
    +				InstantiationUtil.deserializeObject(stateHandles.iterator().next().openInputStream(), userCodeClassLoader);
    +
    +		for (Map.Entry<String, KvStateSnapshot<K, ?, ?, ?>> nameToState : namedStates.entrySet()) {
    +
    +			KvStateSnapshot<K, ?, ?, ?> genericSnapshot = nameToState.getValue();
    +
    +			final RestoredState restoredState;
    +
    +			if (genericSnapshot instanceof AbstractMemStateSnapshot) {
    +
    +				AbstractMemStateSnapshot<K, ?, ?, ?, ?> stateSnapshot =
    +						(AbstractMemStateSnapshot<K, ?, ?, ?, ?>) nameToState.getValue();
    +
    +				restoredState = restoreHeapState(stateSnapshot);
    +
    +			} else if (genericSnapshot instanceof AbstractFsStateSnapshot) {
    +
    +				AbstractFsStateSnapshot<K, ?, ?, ?, ?> stateSnapshot =
    +						(AbstractFsStateSnapshot<K, ?, ?, ?, ?>) nameToState.getValue();
    +				restoredState = restoreFsState(stateSnapshot);
    +			} else {
    +				throw new IllegalStateException("Unknown state: " + genericSnapshot);
    +			}
    +
    +			Map rawResultMap = restoredState.getRawResultMap();
    +			TypeSerializer<?> namespaceSerializer = restoredState.getNamespaceSerializer();
    +			TypeSerializer<?> stateSerializer = restoredState.getStateSerializer();
    +
    +			if (namespaceSerializer instanceof VoidSerializer) {
    +				namespaceSerializer = VoidNamespaceSerializer.INSTANCE;
    +			}
    +
    +			Map nullNameSpaceFix = (Map) rawResultMap.remove(null);
    +
    +			if (null != nullNameSpaceFix) {
    +				rawResultMap.put(VoidNamespace.INSTANCE, nullNameSpaceFix);
    +			}
    +
    +			StateTable<K, ?, ?> stateTable = new StateTable<>(stateSerializer, namespaceSerializer, keyGroupRange);
    +			stateTable.getState().set(0, rawResultMap);
    +
    +			// add named state to the backend
    +			getStateTables().put(nameToState.getKey(), stateTable);
    +		}
    +	}
    +
    +	private RestoredState restoreHeapState(AbstractMemStateSnapshot<K, ?, ?, ?, ?> stateSnapshot) throws IOException {
    +		return new RestoredState(
    +				stateSnapshot.deserialize(),
    +				stateSnapshot.getNamespaceSerializer(),
    +				stateSnapshot.getStateSerializer());
    +	}
    +
    +	private RestoredState restoreFsState(AbstractFsStateSnapshot<K, ?, ?, ?, ?> stateSnapshot) throws IOException {
    +		FileSystem fs = stateSnapshot.getFilePath().getFileSystem();
    +		//TODO register closeable to support fast cancelation?
    +		try (FSDataInputStream inStream = fs.open(stateSnapshot.getFilePath())) {
    +
    +			DataInputViewStreamWrapper inView = new DataInputViewStreamWrapper(inStream);
    +
    +			final int numKeys = inView.readInt();
    --- End diff --
    
    Minor comment: It seems this variable actually means `numNamespaces`.


> Implement savepoint backwards compatibility 1.1 -> 1.2
> ------------------------------------------------------
>
>                 Key: FLINK-5041
>                 URL: https://issues.apache.org/jira/browse/FLINK-5041
>             Project: Flink
>          Issue Type: New Feature
>          Components: State Backends, Checkpointing
>    Affects Versions: 1.2.0
>            Reporter: Stefan Richter
>            Assignee: Stefan Richter
>
> This issue tracks the implementation of backwards compatibility between Flink 1.1 and 1.2 releases.
> This task subsumes:
> - Converting old savepoints to new savepoints, including a conversion of state handles to their new replacement.
> - Converting keyed state from old backend implementations to their new counterparts.
> - Converting operator and function state for all changed operators.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)