You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/05/17 07:03:52 UTC
[GitHub] [flink] Vancior opened a new pull request, #19743: [FLINK-27657][python] Implement remote operator state backend in PyFlink
Vancior opened a new pull request, #19743:
URL: https://github.com/apache/flink/pull/19743
## What is the purpose of the change
This PR implements remote operator state backend in PyFlink, which is for supporting broadcast state.
## Brief change log
- Added broadcast state related interfaces
- Create `RemoteOperatorStateBackend`, reusing existing map state runtime and caching handler
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dianfu commented on a diff in pull request #19743: [FLINK-27657][python] Implement remote operator state backend in PyFlink
Posted by GitBox <gi...@apache.org>.
dianfu commented on code in PR #19743:
URL: https://github.com/apache/flink/pull/19743#discussion_r876949022
##########
flink-python/pyflink/datastream/state.py:
##########
@@ -281,6 +283,110 @@ def __iter__(self) -> Iterator[K]:
return iter(self.keys())
+class ReadOnlyBroadcastState(State, Generic[K, V]):
+ """
+ A read-only view of the :class:`BroadcastState`.
+ Although read-only, the user code should not modify the value returned by the :meth:`get` or the
+ items returned by :meth:`items`, as this can lead to inconsistent states. The reason for this is
+ that we do not create extra copies of the elements for performance reasons.
+ """
+
+ @abstractmethod
+ def get(self, key: K) -> V:
+ """
+ Returns the current value associated with the given key.
+ """
+ pass
+
+ @abstractmethod
+ def contains(self, key: K) -> bool:
+ """
+ Returns whether there exists the given mapping.
+ """
+ pass
+
+ @abstractmethod
+ def items(self) -> Iterable[Tuple[K, V]]:
+ """
+ Returns all the mappings in the state.
+ """
+ pass
+
+ @abstractmethod
+ def keys(self) -> Iterable[K]:
Review Comment:
The methods in this class are not the same as the methods in Java `ReadOnlyBroadcastState`. Is this intended? If so, could you explain it a bit? This class is a public class and so we need to be careful of it.
##########
flink-python/pyflink/datastream/state.py:
##########
@@ -281,6 +283,110 @@ def __iter__(self) -> Iterator[K]:
return iter(self.keys())
+class ReadOnlyBroadcastState(State, Generic[K, V]):
+ """
+ A read-only view of the :class:`BroadcastState`.
+ Although read-only, the user code should not modify the value returned by the :meth:`get` or the
+ items returned by :meth:`items`, as this can lead to inconsistent states. The reason for this is
+ that we do not create extra copies of the elements for performance reasons.
+ """
+
+ @abstractmethod
+ def get(self, key: K) -> V:
+ """
+ Returns the current value associated with the given key.
+ """
+ pass
+
+ @abstractmethod
+ def contains(self, key: K) -> bool:
+ """
+ Returns whether there exists the given mapping.
+ """
+ pass
+
+ @abstractmethod
+ def items(self) -> Iterable[Tuple[K, V]]:
+ """
+ Returns all the mappings in the state.
+ """
+ pass
+
+ @abstractmethod
+ def keys(self) -> Iterable[K]:
+ """
+ Returns all the keys in the state.
+ """
+ pass
+
+ @abstractmethod
+ def values(self) -> Iterable[V]:
+ """
+ Returns all the values in the state.
+ """
+ pass
+
+ @abstractmethod
+ def is_empty(self) -> bool:
+ """
+ Returns true if this state contains no key-value mappings, otherwise false.
+ """
+ pass
+
+ def __getitem__(self, key: K) -> V:
+ return self.get(key)
+
+ def __contains__(self, key: K) -> bool:
+ return self.contains(key)
+
+ def __iter__(self) -> Iterator[K]:
+ return iter(self.keys())
+
+
+class BroadcastState(ReadOnlyBroadcastState[K, V]):
Review Comment:
Ditto. Why the APIs in this class aren't the same as the Java one?
##########
flink-python/pyflink/datastream/__init__.py:
##########
@@ -141,6 +141,11 @@
- :class:`state.AggregatingState`:
Interface for aggregating state, based on an :class:`AggregateFunction`. Elements that are
added to this type of state will be eagerly pre-aggregated using a given AggregateFunction.
+ - :class:`state.BroadcastState`:
+ A type of state that can be created to store the state of a :class:`BroadcastStream`. This
+ state assumes that the same elements are sent to all instances of an operator.
+ - :class:`state.ReadOnlyBroadcastState`:
Review Comment:
I guess users don't need to import this class manually and so there is no need to expose it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Vancior commented on a diff in pull request #19743: [FLINK-27657][python] Implement remote operator state backend in PyFlink
Posted by GitBox <gi...@apache.org>.
Vancior commented on code in PR #19743:
URL: https://github.com/apache/flink/pull/19743#discussion_r877772504
##########
flink-python/src/main/java/org/apache/flink/streaming/api/runners/python/beam/state/BeamStateHandler.java:
##########
@@ -23,6 +23,35 @@
/** Interface for doing actual operations on Flink state based on {@link BeamFnApi.StateRequest}. */
public interface BeamStateHandler<S> {
- BeamFnApi.StateResponse.Builder handle(BeamFnApi.StateRequest request, S state)
+ /**
+ * Dispatches {@link BeamFnApi.StateRequest} to different handle functions base on request case.
+ */
+ default BeamFnApi.StateResponse.Builder handle(BeamFnApi.StateRequest request, S state)
Review Comment:
That's a good point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19743: [FLINK-27657][python] Implement remote operator state backend in PyFlink
Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #19743:
URL: https://github.com/apache/flink/pull/19743#issuecomment-1128493103
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2d778eea8d0b2f6c41fd92e789e0149a37473d71",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d778eea8d0b2f6c41fd92e789e0149a37473d71",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2d778eea8d0b2f6c41fd92e789e0149a37473d71 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Vancior commented on a diff in pull request #19743: [FLINK-27657][python] Implement remote operator state backend in PyFlink
Posted by GitBox <gi...@apache.org>.
Vancior commented on code in PR #19743:
URL: https://github.com/apache/flink/pull/19743#discussion_r877770129
##########
flink-python/pyflink/datastream/state.py:
##########
@@ -281,6 +283,110 @@ def __iter__(self) -> Iterator[K]:
return iter(self.keys())
+class ReadOnlyBroadcastState(State, Generic[K, V]):
+ """
+ A read-only view of the :class:`BroadcastState`.
+ Although read-only, the user code should not modify the value returned by the :meth:`get` or the
+ items returned by :meth:`items`, as this can lead to inconsistent states. The reason for this is
+ that we do not create extra copies of the elements for performance reasons.
+ """
+
+ @abstractmethod
+ def get(self, key: K) -> V:
+ """
+ Returns the current value associated with the given key.
+ """
+ pass
+
+ @abstractmethod
+ def contains(self, key: K) -> bool:
+ """
+ Returns whether there exists the given mapping.
+ """
+ pass
+
+ @abstractmethod
+ def items(self) -> Iterable[Tuple[K, V]]:
+ """
+ Returns all the mappings in the state.
+ """
+ pass
+
+ @abstractmethod
+ def keys(self) -> Iterable[K]:
Review Comment:
We don't have the exactly same interfaces for `MapState`, which might be more intuitive for python users, so I just align `BroadcastState` with `MapState`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Vancior commented on a diff in pull request #19743: [FLINK-27657][python] Implement remote operator state backend in PyFlink
Posted by GitBox <gi...@apache.org>.
Vancior commented on code in PR #19743:
URL: https://github.com/apache/flink/pull/19743#discussion_r877773328
##########
flink-python/pyflink/datastream/__init__.py:
##########
@@ -141,6 +141,11 @@
- :class:`state.AggregatingState`:
Interface for aggregating state, based on an :class:`AggregateFunction`. Elements that are
added to this type of state will be eagerly pre-aggregated using a given AggregateFunction.
+ - :class:`state.BroadcastState`:
+ A type of state that can be created to store the state of a :class:`BroadcastStream`. This
+ state assumes that the same elements are sent to all instances of an operator.
+ - :class:`state.ReadOnlyBroadcastState`:
Review Comment:
Might be convenient for finding what operations can be done?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangXingBo closed pull request #19743: [FLINK-27657][python] Implement remote operator state backend in PyFlink
Posted by GitBox <gi...@apache.org>.
HuangXingBo closed pull request #19743: [FLINK-27657][python] Implement remote operator state backend in PyFlink
URL: https://github.com/apache/flink/pull/19743
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangXingBo commented on a diff in pull request #19743: [FLINK-27657][python] Implement remote operator state backend in PyFlink
Posted by GitBox <gi...@apache.org>.
HuangXingBo commented on code in PR #19743:
URL: https://github.com/apache/flink/pull/19743#discussion_r874597805
##########
flink-python/pyflink/datastream/functions.py:
##########
@@ -119,6 +119,40 @@ def get_aggregating_state(
pass
+class OperatorStateStore(ABC):
+ """
+ Interface for getting operator states. Currently, only :class:`~state.BroadcastState` is
+ supported.
+ .. versionadded:: 1.16.0
+ """
+
+ @abstractmethod
+ def get_broadcast_state(self, state_descriptor: MapStateDescriptor) -> BroadcastState:
+ """
+ Fetches the :class:`~state.BroadcastState` described by :class:`~state.MapStateDescriptor`,
+ which has read/write access to the broadcast operator state.
+ """
+ pass
+
+ @abstractmethod
+ def get_read_only_broadcast_state(
Review Comment:
I don't think `OperatorStateStore` should have this method. Whether `BroadcastState` is readonly should depend on whether it is in `process_element` or `process_broad_cast_element`
##########
flink-python/pyflink/datastream/functions.py:
##########
@@ -119,6 +119,40 @@ def get_aggregating_state(
pass
+class OperatorStateStore(ABC):
+ """
+ Interface for getting operator states. Currently, only :class:`~state.BroadcastState` is
+ supported.
+ .. versionadded:: 1.16.0
+ """
+
+ @abstractmethod
+ def get_broadcast_state(self, state_descriptor: MapStateDescriptor) -> BroadcastState:
+ """
+ Fetches the :class:`~state.BroadcastState` described by :class:`~state.MapStateDescriptor`,
+ which has read/write access to the broadcast operator state.
+ """
+ pass
+
+ @abstractmethod
+ def get_read_only_broadcast_state(
+ self, state_descriptor: MapStateDescriptor
+ ) -> ReadOnlyBroadcastState:
+ """
+ Fetches the :class:`~state.ReadOnlyBroadcastState` described by
+ :class:`~state.MapStateDescriptor`, which only has read access to the broadcast operator
+ state.
+ """
+ pass
+
+ @abstractmethod
+ def commit(self):
Review Comment:
I think the `commit` method is related to your underlying implementation of communication, the base class `OperatorStateStore` should not include this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Vancior commented on a diff in pull request #19743: [FLINK-27657][python] Implement remote operator state backend in PyFlink
Posted by GitBox <gi...@apache.org>.
Vancior commented on code in PR #19743:
URL: https://github.com/apache/flink/pull/19743#discussion_r877773328
##########
flink-python/pyflink/datastream/__init__.py:
##########
@@ -141,6 +141,11 @@
- :class:`state.AggregatingState`:
Interface for aggregating state, based on an :class:`AggregateFunction`. Elements that are
added to this type of state will be eagerly pre-aggregated using a given AggregateFunction.
+ - :class:`state.BroadcastState`:
+ A type of state that can be created to store the state of a :class:`BroadcastStream`. This
+ state assumes that the same elements are sent to all instances of an operator.
+ - :class:`state.ReadOnlyBroadcastState`:
Review Comment:
Might be convenient for type hint?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] HuangXingBo commented on a diff in pull request #19743: [FLINK-27657][python] Implement remote operator state backend in PyFlink
Posted by GitBox <gi...@apache.org>.
HuangXingBo commented on code in PR #19743:
URL: https://github.com/apache/flink/pull/19743#discussion_r876562559
##########
flink-python/pyflink/datastream/functions.py:
##########
@@ -119,6 +118,22 @@ def get_aggregating_state(
pass
+class OperatorStateStore(ABC):
Review Comment:
What about moving this class to the `state.py`?
##########
flink-python/src/main/java/org/apache/flink/streaming/api/runners/python/beam/state/BeamStateHandler.java:
##########
@@ -23,6 +23,35 @@
/** Interface for doing actual operations on Flink state based on {@link BeamFnApi.StateRequest}. */
public interface BeamStateHandler<S> {
- BeamFnApi.StateResponse.Builder handle(BeamFnApi.StateRequest request, S state)
+ /**
+ * Dispatches {@link BeamFnApi.StateRequest} to different handle functions base on request case.
+ */
+ default BeamFnApi.StateResponse.Builder handle(BeamFnApi.StateRequest request, S state)
Review Comment:
I wonder if it would be clearer if we introduce an abstract class of `AbstractBeamStateHandler`. Personally, I don't like the form of default interface methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Vancior commented on a diff in pull request #19743: [FLINK-27657][python] Implement remote operator state backend in PyFlink
Posted by GitBox <gi...@apache.org>.
Vancior commented on code in PR #19743:
URL: https://github.com/apache/flink/pull/19743#discussion_r877772151
##########
flink-python/pyflink/datastream/functions.py:
##########
@@ -119,6 +118,22 @@ def get_aggregating_state(
pass
+class OperatorStateStore(ABC):
Review Comment:
I'm just following where `KeyedStateStore` is, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Vancior commented on a diff in pull request #19743: [FLINK-27657][python] Implement remote operator state backend in PyFlink
Posted by GitBox <gi...@apache.org>.
Vancior commented on code in PR #19743:
URL: https://github.com/apache/flink/pull/19743#discussion_r877785593
##########
flink-python/pyflink/datastream/state.py:
##########
@@ -281,6 +283,110 @@ def __iter__(self) -> Iterator[K]:
return iter(self.keys())
+class ReadOnlyBroadcastState(State, Generic[K, V]):
+ """
+ A read-only view of the :class:`BroadcastState`.
+ Although read-only, the user code should not modify the value returned by the :meth:`get` or the
+ items returned by :meth:`items`, as this can lead to inconsistent states. The reason for this is
+ that we do not create extra copies of the elements for performance reasons.
+ """
+
+ @abstractmethod
+ def get(self, key: K) -> V:
+ """
+ Returns the current value associated with the given key.
+ """
+ pass
+
+ @abstractmethod
+ def contains(self, key: K) -> bool:
+ """
+ Returns whether there exists the given mapping.
+ """
+ pass
+
+ @abstractmethod
+ def items(self) -> Iterable[Tuple[K, V]]:
+ """
+ Returns all the mappings in the state.
+ """
+ pass
+
+ @abstractmethod
+ def keys(self) -> Iterable[K]:
Review Comment:
Since users already have `items()` interface, `keys()` and `values()` just making it more like `dict` in python, without not much extra code giving that the underlying state runtime implementation is actually map state too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org