You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by StephanEwen <gi...@git.apache.org> on 2015/04/30 22:24:57 UTC
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/643
[streaming] New Source and state checkpointing interfaces
These interfaces allow streaming sources/operators to interact with the state checkpointing in a more precise manner.
The interfaces are so far intended mainly for internal state handling (sources, window operators).
An open question what state interface we want to expose to the user code such that, on scale-in and scale-out, the state repartitions consistently with the data partitions.
- For groupByKey operations, we could scope the state by key.
- For all other operations, state may be repartitioned in a user-defined manner. This would give no guarantees about alignment with partitions. Think of a pipeline `partionBy -> map()`. The state accumulated in the `map()` function is partitioned by a key, but the state entries may not be able to be linked to a certain key or key-hash value.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StephanEwen/incubator-flink cp_interfaces
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/643.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #643
----
commit 76d63d5b03643d3c6e51a882eb9427e421806342
Author: Stephan Ewen <se...@apache.org>
Date: 2015-04-30T20:05:27Z
[streaming] New Source and state checkpointing interfaces that allow operations to interact with the state checkpointing in a more precise manner.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/643#issuecomment-99063478
That is a good question. If we assume for now that tasking an incremental versus a full snapshot is a decision that each task takes individually, then I think it would work, as the checkpointing code does not see any difference.
The difference would be encoded in the state handle, which would have multiple fields, like `latestFullState` and `List<StateDelta>`. The operator can then reconstruct its full state form that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/643#discussion_r29582102
--- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/functions/source/StreamSource.java ---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.functions.source;
+
+/**
+ * Base interface for all stream data sources in Flink. The contract of a stream source
+ * is similar to an iterator - it is consumed as in the following pseudo code:
+ *
+ * <pre>{@code
+ * StreamSource<T> source = ...;
+ * Collector<T> out = ...;
+ * while (!source.reachedEnd()) {
+ * out.collect(source.next());
+ * }
+ * }
+ * </pre>
+ *
+ * <b>Note about blocking behavior</b>
+ * <p>This implementations of the methods in the stream sources must have certain guarantees about
+ * blocking behavior. One of the two characteristics must be fulfilled.</p>
+ * <ul>
+ * <li>The methods must react to thread interrupt calls and break out of blocking calls with
+ * an {@link InterruptedException}.</li>
+ * <li>The method may ignore interrupt calls and/or swallow InterruptedExceptions, if it is guaranteed
+ * that the method returns quasi immediately irrespectively of the input. This is true for example
+ * for file streams, where the call is guaranteed to return after a very short I/O delay in
+ * the order of milliseconds.</li>
+ * </ul>
+ *
+ * @param <T> The type of the records produced by this source.
+ */
+public interface StreamSource<T> {
+
+ /**
+ * Checks whether the stream has reached its end.
+ *
+ * <p>This method must obey the contract about blocking behavior declared in the
+ * description of this class.</p>
+ *
+ * @return True, if the end of the stream has been reached, false if more data is available.
+ *
+ * @throws InterruptedException The calling thread may be interrupted to pull the function out of this
+ * method during checkpoints.
+ * @throws Exception Any other exception that is thrown causes the source to fail and results in failure of
+ * the streaming program, or triggers recovery, depending on the program setup.
+ */
+ boolean reachedEnd() throws Exception;
+
--- End diff --
This is in line with the batch InputFormats, that is the only reason.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by senorcarbone <gi...@git.apache.org>.
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/643#issuecomment-98757999
I like the interfaces, they are quite precise. To sum up (and see if I get this right), with `Checkpointed` we simply guarantee that an operator will not get invoked while state is persisted unless it implements `CheckpointedAsynchronously`. In that case the operator should instead send a copy of its state.
Regarding reconfiguration with arbitrary partitioning, a scheme that comes into my mind right now is to let functions to lazily request state on demand (without priorly injecting it) when they need to access it since we can't possible preallocate it. The first time they will try to access specific keys for example, only the associated state with these keys will be retrieved from the persistent recovery checkpoint. This way we let the reconfigured partitioners to decide on how state will be eventually allocated to instances.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by mbalassi <gi...@git.apache.org>.
Github user mbalassi commented on the pull request:
https://github.com/apache/flink/pull/643#issuecomment-98637993
The changes seem reasonable, added comments inline.
As for the state elasticity my gut feeling is that we should be OK if we get the grouped case right. Besides that we do not provide partitioning guarantees for the records anyway (`partitionBy` is protected and only used when `groupBy` is called in `DataStream`).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/643#issuecomment-98703915
@aljoscha I think this is orthogonal to the changes here. What you suggest is a more elaborate version of the `StateHandle`. For that, I totally agree.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen closed the pull request at:
https://github.com/apache/flink/pull/643
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/643#issuecomment-103116083
Manually merged!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by mbalassi <gi...@git.apache.org>.
Github user mbalassi commented on a diff in the pull request:
https://github.com/apache/flink/pull/643#discussion_r29571516
--- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/checkpoint/Checkpointed.java ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.checkpoint;
+
+import org.apache.flink.runtime.state.OperatorState;
+
+/**
+ * This method must be implemented by functions that have state that needs to be
+ * checkpointed. The functions get a call whenever a checkpoint should take place
+ * and return a snapshot of their state, which will be checkpointed.
+ *
+ * <p>This interface marks a function as <i>synchronously</i> checkpointed. While the
+ * state is written, the function is not called, so the function needs not return a
+ * copy of its state, but may return a reference to its state. Functions that can
+ * continue to work and mutate the state, even while the state snapshot is being accessed,
+ * can implement the {@link org.apache.flink.streaming.api.checkpoint.CheckpointedAsynchronously}
+ * interface.</p>
+ */
+public interface Checkpointed {
+
+ /**
+ * Gets the current operator state as a checkpoint. The state must reflect all operations
+ * from all prior operations if this function.
+ *
+ * @param checkpointId The ID of the checkpoint.
+ * @param checkpointTimestamp The timestamp of the checkpoint, as derived by
+ * System.currentTimeMillis() on the JobManager.
+ *
+ * @return A snapshot of the operator state.
+ *
+ * @throws Exception Thrown if the creation of the state object failed. This causes the
+ * checkpoint to fail. The system may decide to fail the operation (and trigger
+ * recovery), or to discard this checkpoint attempt and to continue running
+ * and to try again with the next checkpoint attempt.
+ */
+ OperatorState<?> snapshotState(long checkpointId, long checkpointTimestamp) throws Exception;
+}
--- End diff --
If the `checkpointTimestamp` is also derived at the JobManager why do we need both here? Maybe I am missing something. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/643#issuecomment-98760015
Concerning synchronous/asynchronous checkpointing, you are right. For asynchronous checkpointing, you may also use some copy-on-write techniques, rather than doing an eager copy.
For reconfiguring the state, we need still a way to have more fine grained handles to the state, to be able to split it in a way that is consistent with re-configuring the partitioner functions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/643#discussion_r29582495
--- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/checkpoint/Checkpointed.java ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.checkpoint;
+
+import org.apache.flink.runtime.state.OperatorState;
+
+/**
+ * This method must be implemented by functions that have state that needs to be
+ * checkpointed. The functions get a call whenever a checkpoint should take place
+ * and return a snapshot of their state, which will be checkpointed.
+ *
+ * <p>This interface marks a function as <i>synchronously</i> checkpointed. While the
+ * state is written, the function is not called, so the function needs not return a
+ * copy of its state, but may return a reference to its state. Functions that can
+ * continue to work and mutate the state, even while the state snapshot is being accessed,
+ * can implement the {@link org.apache.flink.streaming.api.checkpoint.CheckpointedAsynchronously}
+ * interface.</p>
+ */
+public interface Checkpointed {
+
+ /**
+ * Gets the current operator state as a checkpoint. The state must reflect all operations
+ * from all prior operations if this function.
+ *
+ * @param checkpointId The ID of the checkpoint.
+ * @param checkpointTimestamp The timestamp of the checkpoint, as derived by
+ * System.currentTimeMillis() on the JobManager.
+ *
+ * @return A snapshot of the operator state.
+ *
+ * @throws Exception Thrown if the creation of the state object failed. This causes the
+ * checkpoint to fail. The system may decide to fail the operation (and trigger
+ * recovery), or to discard this checkpoint attempt and to continue running
+ * and to try again with the next checkpoint attempt.
+ */
+ OperatorState<?> snapshotState(long checkpointId, long checkpointTimestamp) throws Exception;
+}
--- End diff --
The checkpoint ID and the timestamp are also different. The ID is random unique and the timstamp is time
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/643#issuecomment-99073001
Sorry, my bad, I confused the layers.
These interfaces do not sufficiently describe things for incremental checkpointing. We may have to introduce additional interfaces for functions or operators that support incremental snapshotting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by mbalassi <gi...@git.apache.org>.
Github user mbalassi commented on a diff in the pull request:
https://github.com/apache/flink/pull/643#discussion_r29571645
--- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/functions/source/StreamSource.java ---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.functions.source;
+
+/**
+ * Base interface for all stream data sources in Flink. The contract of a stream source
+ * is similar to an iterator - it is consumed as in the following pseudo code:
+ *
+ * <pre>{@code
+ * StreamSource<T> source = ...;
+ * Collector<T> out = ...;
+ * while (!source.reachedEnd()) {
+ * out.collect(source.next());
+ * }
+ * }
+ * </pre>
+ *
+ * <b>Note about blocking behavior</b>
+ * <p>This implementations of the methods in the stream sources must have certain guarantees about
+ * blocking behavior. One of the two characteristics must be fulfilled.</p>
+ * <ul>
+ * <li>The methods must react to thread interrupt calls and break out of blocking calls with
+ * an {@link InterruptedException}.</li>
+ * <li>The method may ignore interrupt calls and/or swallow InterruptedExceptions, if it is guaranteed
+ * that the method returns quasi immediately irrespectively of the input. This is true for example
+ * for file streams, where the call is guaranteed to return after a very short I/O delay in
+ * the order of milliseconds.</li>
+ * </ul>
+ *
+ * @param <T> The type of the records produced by this source.
+ */
+public interface StreamSource<T> {
+
+ /**
+ * Checks whether the stream has reached its end.
+ *
+ * <p>This method must obey the contract about blocking behavior declared in the
+ * description of this class.</p>
+ *
+ * @return True, if the end of the stream has been reached, false if more data is available.
+ *
+ * @throws InterruptedException The calling thread may be interrupted to pull the function out of this
+ * method during checkpoints.
+ * @throws Exception Any other exception that is thrown causes the source to fail and results in failure of
+ * the streaming program, or triggers recovery, depending on the program setup.
+ */
+ boolean reachedEnd() throws Exception;
+
--- End diff --
Why did you choose to explicitly differ from the standard `hasNext()` name?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/643#issuecomment-98690440
I'm afraid we need to be a bit more strict here. We need to partition the state by key, to make it more manageable. Therefore, we should only allow the user to use state if he provides a key, for example using the (non-existant) keyBy(). Then we can partition the data and the operators know for which part of the state they are responsible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/643#discussion_r29582309
--- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/checkpoint/Checkpointed.java ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.checkpoint;
+
+import org.apache.flink.runtime.state.OperatorState;
+
+/**
+ * This method must be implemented by functions that have state that needs to be
+ * checkpointed. The functions get a call whenever a checkpoint should take place
+ * and return a snapshot of their state, which will be checkpointed.
+ *
+ * <p>This interface marks a function as <i>synchronously</i> checkpointed. While the
+ * state is written, the function is not called, so the function needs not return a
+ * copy of its state, but may return a reference to its state. Functions that can
+ * continue to work and mutate the state, even while the state snapshot is being accessed,
+ * can implement the {@link org.apache.flink.streaming.api.checkpoint.CheckpointedAsynchronously}
+ * interface.</p>
+ */
+public interface Checkpointed {
+
+ /**
+ * Gets the current operator state as a checkpoint. The state must reflect all operations
+ * from all prior operations if this function.
+ *
+ * @param checkpointId The ID of the checkpoint.
+ * @param checkpointTimestamp The timestamp of the checkpoint, as derived by
+ * System.currentTimeMillis() on the JobManager.
+ *
+ * @return A snapshot of the operator state.
+ *
+ * @throws Exception Thrown if the creation of the state object failed. This causes the
+ * checkpoint to fail. The system may decide to fail the operation (and trigger
+ * recovery), or to discard this checkpoint attempt and to continue running
+ * and to try again with the next checkpoint attempt.
+ */
+ OperatorState<?> snapshotState(long checkpointId, long checkpointTimestamp) throws Exception;
+}
--- End diff --
The timestamp is the time when the checkpoint was started, assigned by the JobManager, part of the messages that the tasks receive.
All barriers emitted into the stream should to have the same timestamp, if we do consistent "source-assigned event time" - it makes triggering of time-tiggered windows much easier in operators.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by mbalassi <gi...@git.apache.org>.
Github user mbalassi commented on the pull request:
https://github.com/apache/flink/pull/643#issuecomment-103115919
I think these commits are in. @StephanEwen could you please close the PR manually?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by gyfora <gi...@git.apache.org>.
Github user gyfora commented on the pull request:
https://github.com/apache/flink/pull/643#issuecomment-98964566
Do you think these interfaces will work if at any point we want to do incremental snapshots?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [streaming] New Source and state checkpointing...
Posted by aljoscha <gi...@git.apache.org>.
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/643#issuecomment-98690522
Also, the key would be a property of a DataStream, and other operations could also use this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---