You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by StefanRRichter <gi...@git.apache.org> on 2016/12/05 20:15:20 UTC
[GitHub] flink pull request #2948: [FLINK-5265] Introduce state handle replication mo...
GitHub user StefanRRichter opened a pull request:
https://github.com/apache/flink/pull/2948
[FLINK-5265] Introduce state handle replication mode for CheckpointCoordinator
Currently, the ``CheckpointCoordinator`` only supports repartitioning of ``OperatorStateHandle``s based on a split-and-distribute strategy. For future state types, such as broadcast or union state, we need a different repartitioning method that allows for replicating state handles to all subtasks.
This functionality is introduced with this PR and represents the first step on the way to implementing broadcast and union states.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StefanRRichter/flink broadcast-op-state
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2948.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2948
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2948: [FLINK-5265] Introduce state handle replication mode for ...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2948
Thanks for your work! \U0001f44d I merged, could you please close this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2948: [FLINK-5265] Introduce state handle replication mo...
Posted by StefanRRichter <gi...@git.apache.org>.
Github user StefanRRichter commented on a diff in the pull request:
https://github.com/apache/flink/pull/2948#discussion_r95991932
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorStateCheckpointOutputStream.java ---
@@ -66,8 +66,10 @@ OperatorStateHandle closeAndGetHandle() throws IOException {
startNewPartition();
}
- Map<String, long[]> offsetsMap = new HashMap<>(1);
- offsetsMap.put(DefaultOperatorStateBackend.DEFAULT_OPERATOR_STATE_NAME, partitionOffsets.toArray());
+ Map<String, OperatorStateHandle.StateMetaInfo> offsetsMap = new HashMap<>(1);
+ OperatorStateHandle.StateMetaInfo metaInfo =
+ new OperatorStateHandle.StateMetaInfo(partitionOffsets.toArray(), OperatorStateHandle.Mode.BROADCAST);
--- End diff --
Yes, that is correct :-). Will change this as suggested.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #2948: [FLINK-5265] Introduce state handle replication mode for ...
Posted by StefanRRichter <gi...@git.apache.org>.
Github user StefanRRichter commented on the issue:
https://github.com/apache/flink/pull/2948
CC @aljoscha @StephanEwen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2948: [FLINK-5265] Introduce state handle replication mo...
Posted by StefanRRichter <gi...@git.apache.org>.
Github user StefanRRichter closed the pull request at:
https://github.com/apache/flink/pull/2948
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #2948: [FLINK-5265] Introduce state handle replication mo...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/2948#discussion_r95979296
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/state/OperatorStateCheckpointOutputStream.java ---
@@ -66,8 +66,10 @@ OperatorStateHandle closeAndGetHandle() throws IOException {
startNewPartition();
}
- Map<String, long[]> offsetsMap = new HashMap<>(1);
- offsetsMap.put(DefaultOperatorStateBackend.DEFAULT_OPERATOR_STATE_NAME, partitionOffsets.toArray());
+ Map<String, OperatorStateHandle.StateMetaInfo> offsetsMap = new HashMap<>(1);
+ OperatorStateHandle.StateMetaInfo metaInfo =
+ new OperatorStateHandle.StateMetaInfo(partitionOffsets.toArray(), OperatorStateHandle.Mode.BROADCAST);
--- End diff --
I think this should be `SPLIT_REDISTRIBUTE` to conform to the old behaviour.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---