You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/03/11 07:08:27 UTC
[GitHub] [flink] Vancior opened a new pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Vancior opened a new pull request #19054:
URL: https://github.com/apache/flink/pull/19054
## What is the purpose of the change
This pull request implements `WindowedStream.reduce` in Python DataStream API.
## Verifying this change
This change added tests and can be verified as follows:
- Added integration tests in test_data_stream.py
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (Sphinx doc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19054:
URL: https://github.com/apache/flink/pull/19054#issuecomment-1064837015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32889",
"triggerID" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"triggerType" : "PUSH"
}, {
"hash" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32989",
"triggerID" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"triggerType" : "PUSH"
}, {
"hash" : "591c73b1c352667583b076abb19e8f259fbb4d38",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "591c73b1c352667583b076abb19e8f259fbb4d38",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d95b4714a85b0435ef78757e200c2ba92ee22e8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32989)
* 591c73b1c352667583b076abb19e8f259fbb4d38 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19054:
URL: https://github.com/apache/flink/pull/19054#issuecomment-1064837015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32889",
"triggerID" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"triggerType" : "PUSH"
}, {
"hash" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32989",
"triggerID" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"triggerType" : "PUSH"
}, {
"hash" : "591c73b1c352667583b076abb19e8f259fbb4d38",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33140",
"triggerID" : "591c73b1c352667583b076abb19e8f259fbb4d38",
"triggerType" : "PUSH"
}, {
"hash" : "d58e60288dc587f40e57565a76973488ea692ad2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33204",
"triggerID" : "d58e60288dc587f40e57565a76973488ea692ad2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 591c73b1c352667583b076abb19e8f259fbb4d38 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33140)
* d58e60288dc587f40e57565a76973488ea692ad2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33204)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19054:
URL: https://github.com/apache/flink/pull/19054#issuecomment-1064837015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32889",
"triggerID" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"triggerType" : "PUSH"
}, {
"hash" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32989",
"triggerID" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d95b4714a85b0435ef78757e200c2ba92ee22e8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32989)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dianfu closed pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
dianfu closed pull request #19054:
URL: https://github.com/apache/flink/pull/19054
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19054:
URL: https://github.com/apache/flink/pull/19054#issuecomment-1064837015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32889",
"triggerID" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"triggerType" : "PUSH"
}, {
"hash" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32989",
"triggerID" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"triggerType" : "PUSH"
}, {
"hash" : "591c73b1c352667583b076abb19e8f259fbb4d38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33140",
"triggerID" : "591c73b1c352667583b076abb19e8f259fbb4d38",
"triggerType" : "PUSH"
}, {
"hash" : "d58e60288dc587f40e57565a76973488ea692ad2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33204",
"triggerID" : "d58e60288dc587f40e57565a76973488ea692ad2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d58e60288dc587f40e57565a76973488ea692ad2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33204)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19054:
URL: https://github.com/apache/flink/pull/19054#issuecomment-1064837015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32889",
"triggerID" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"triggerType" : "PUSH"
}, {
"hash" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32989",
"triggerID" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"triggerType" : "PUSH"
}, {
"hash" : "591c73b1c352667583b076abb19e8f259fbb4d38",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33140",
"triggerID" : "591c73b1c352667583b076abb19e8f259fbb4d38",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7d95b4714a85b0435ef78757e200c2ba92ee22e8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32989)
* 591c73b1c352667583b076abb19e8f259fbb4d38 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33140)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19054:
URL: https://github.com/apache/flink/pull/19054#issuecomment-1064837015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32889",
"triggerID" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c14b7fc09537f4959b8dccc25722140ad2e24ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32889)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dianfu commented on a change in pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
dianfu commented on a change in pull request #19054:
URL: https://github.com/apache/flink/pull/19054#discussion_r828775940
##########
File path: docs/content/docs/dev/datastream/operators/windows.md
##########
@@ -462,6 +462,17 @@ input
.reduce { (v1, v2) => (v1._1, v1._2 + v2._2) }
```
{{< /tab >}}
+{{< tab "Python" >}}
+```python
+input: DataStream = ...
+
+input
Review comment:
```suggestion
input \
```
##########
File path: docs/content/docs/dev/datastream/operators/windows.md
##########
@@ -462,6 +462,17 @@ input
.reduce { (v1, v2) => (v1._1, v1._2 + v2._2) }
Review comment:
Should also update file content.zh/docs/dev/datastream/operators/windows.md
##########
File path: docs/content/docs/dev/datastream/operators/windows.md
##########
@@ -462,6 +462,17 @@ input
.reduce { (v1, v2) => (v1._1, v1._2 + v2._2) }
```
{{< /tab >}}
+{{< tab "Python" >}}
+```python
+input: DataStream = ...
Review comment:
```suggestion
input = ... # type: DataStream
```
##########
File path: docs/content/docs/dev/datastream/operators/windows.md
##########
@@ -462,6 +462,17 @@ input
.reduce { (v1, v2) => (v1._1, v1._2 + v2._2) }
```
{{< /tab >}}
+{{< tab "Python" >}}
+```python
+input: DataStream = ...
+
+input
+ .key_by(<key selector>)
+ .window(<window assigner>)
Review comment:
```suggestion
.window(<window assigner>) \
```
##########
File path: docs/content/docs/dev/datastream/operators/windows.md
##########
@@ -462,6 +462,17 @@ input
.reduce { (v1, v2) => (v1._1, v1._2 + v2._2) }
```
{{< /tab >}}
+{{< tab "Python" >}}
+```python
+input: DataStream = ...
+
+input
+ .key_by(<key selector>)
Review comment:
```suggestion
.key_by(<key selector>) \
```
##########
File path: flink-python/pyflink/datastream/data_stream.py
##########
@@ -1347,8 +1353,58 @@ def allowed_lateness(self, time_ms: int):
self._allowed_lateness = time_ms
return self
+ def reduce(self,
+ reduce_function: Union[Callable, ReduceFunction],
+ window_function: Union[WindowFunction, ProcessWindowFunction] = None,
+ output_type: TypeInformation = None) -> DataStream:
+ """
+ Applies a reduce function to the window. The window function is called for each evaluation
+ of the window for each key individually. The output of the reduce function is interpreted as
+ a regular non-windowed stream.
+
+ This window will try and incrementally aggregate data as much as the window policies
+ permit. For example, tumbling time windows can aggregate the data, meaning that only one
+ element per key is stored. Sliding time windows will aggregate on the granularity of the
+ slide interval, so a few elements are stored per key (one per slide interval). Custom
+ windows may not be able to incrementally aggregate, or may need to store extra values in an
+ aggregation tree.
+
+ Example:
+ ::
+
+ >>> ds.key_by(lambda x: x[1]) \
+ .window(TumblingEventTimeWindow.of(Time.seconds(5))) \
Review comment:
```suggestion
.window(TumblingEventTimeWindows.of(Time.seconds(5))) \
```
##########
File path: flink-python/pyflink/datastream/data_stream.py
##########
@@ -1347,8 +1353,58 @@ def allowed_lateness(self, time_ms: int):
self._allowed_lateness = time_ms
return self
+ def reduce(self,
+ reduce_function: Union[Callable, ReduceFunction],
+ window_function: Union[WindowFunction, ProcessWindowFunction] = None,
+ output_type: TypeInformation = None) -> DataStream:
+ """
+ Applies a reduce function to the window. The window function is called for each evaluation
+ of the window for each key individually. The output of the reduce function is interpreted as
+ a regular non-windowed stream.
+
+ This window will try and incrementally aggregate data as much as the window policies
+ permit. For example, tumbling time windows can aggregate the data, meaning that only one
+ element per key is stored. Sliding time windows will aggregate on the granularity of the
+ slide interval, so a few elements are stored per key (one per slide interval). Custom
+ windows may not be able to incrementally aggregate, or may need to store extra values in an
+ aggregation tree.
+
+ Example:
+ ::
+
+ >>> ds.key_by(lambda x: x[1]) \
+ .window(TumblingEventTimeWindow.of(Time.seconds(5))) \
Review comment:
The format needs to be corrected. Currently, the generated doc looks like the following:
##########
File path: flink-python/pyflink/datastream/data_stream.py
##########
@@ -1347,8 +1353,58 @@ def allowed_lateness(self, time_ms: int):
self._allowed_lateness = time_ms
return self
+ def reduce(self,
+ reduce_function: Union[Callable, ReduceFunction],
+ window_function: Union[WindowFunction, ProcessWindowFunction] = None,
+ output_type: TypeInformation = None) -> DataStream:
+ """
+ Applies a reduce function to the window. The window function is called for each evaluation
+ of the window for each key individually. The output of the reduce function is interpreted as
+ a regular non-windowed stream.
+
+ This window will try and incrementally aggregate data as much as the window policies
+ permit. For example, tumbling time windows can aggregate the data, meaning that only one
+ element per key is stored. Sliding time windows will aggregate on the granularity of the
+ slide interval, so a few elements are stored per key (one per slide interval). Custom
+ windows may not be able to incrementally aggregate, or may need to store extra values in an
+ aggregation tree.
+
+ Example:
+ ::
+
+ >>> ds.key_by(lambda x: x[1]) \
+ .window(TumblingEventTimeWindow.of(Time.seconds(5))) \
+ .reduce(lambda a, b: a[0] + b[0], b[1])
+
+ :param reduce_function: The reduce function.
+ :param window_function: The window function.
+ :param output_type: Type information for the result type of the window function.
+ :return: The data stream that is the result of applying the reduce function to the window.
+
+ .. versionadded:: 1.16
Review comment:
```suggestion
.. versionadded:: 1.16.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #19054:
URL: https://github.com/apache/flink/pull/19054#issuecomment-1064837015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c14b7fc09537f4959b8dccc25722140ad2e24ba UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19054:
URL: https://github.com/apache/flink/pull/19054#issuecomment-1064837015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32889",
"triggerID" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c14b7fc09537f4959b8dccc25722140ad2e24ba Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32889)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19054:
URL: https://github.com/apache/flink/pull/19054#issuecomment-1064837015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32889",
"triggerID" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"triggerType" : "PUSH"
}, {
"hash" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c14b7fc09537f4959b8dccc25722140ad2e24ba Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32889)
* 7d95b4714a85b0435ef78757e200c2ba92ee22e8 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dianfu commented on a change in pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
dianfu commented on a change in pull request #19054:
URL: https://github.com/apache/flink/pull/19054#discussion_r825543942
##########
File path: flink-python/pyflink/datastream/data_stream.py
##########
@@ -1332,6 +1337,55 @@ def allowed_lateness(self, time_ms: int):
self._allowed_lateness = time_ms
return self
+ def reduce(self,
+ reduce_function: Union[Callable, ReduceFunction],
+ window_function: Union[WindowFunction, ProcessWindowFunction] = None,
+ result_type: TypeInformation = None) -> DataStream:
Review comment:
```suggestion
output_type: TypeInformation = None) -> DataStream:
```
Change to output_type to keep it consistent with the other methods?
##########
File path: flink-python/pyflink/datastream/data_stream.py
##########
@@ -1332,6 +1337,55 @@ def allowed_lateness(self, time_ms: int):
self._allowed_lateness = time_ms
return self
+ def reduce(self,
+ reduce_function: Union[Callable, ReduceFunction],
+ window_function: Union[WindowFunction, ProcessWindowFunction] = None,
+ result_type: TypeInformation = None) -> DataStream:
+ """
+ Applies a reduce function to the window. The window function is called for each evaluation of
Review comment:
checksytle issue: line too long (101 > 100 characters)
You can run the following command under flink-python directory for all checkstyle issues:
./dev/lint-python.sh -e flake8,sphinx
##########
File path: flink-python/pyflink/datastream/functions.py
##########
@@ -1004,15 +1010,37 @@ def process(self,
key: KEY,
window: W,
context: InternalWindowContext,
- input_data: Iterable[IN]) -> Iterable[OUT]:
Review comment:
What's the purpose of this change?
##########
File path: flink-python/pyflink/datastream/tests/test_data_stream.py
##########
@@ -794,6 +794,62 @@ def test_time_window(self):
expected = ['(hi,1)', '(hi,1)', '(hi,2)', '(hi,3)']
self.assert_equals_sorted(expected, results)
+ def test_time_window_reduce_passthrough(self):
+ self.env.set_parallelism(1)
Review comment:
It's not necessary to set the parallelism to 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] Vancior commented on a change in pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
Vancior commented on a change in pull request #19054:
URL: https://github.com/apache/flink/pull/19054#discussion_r825552305
##########
File path: flink-python/pyflink/datastream/functions.py
##########
@@ -1004,15 +1010,37 @@ def process(self,
key: KEY,
window: W,
context: InternalWindowContext,
- input_data: Iterable[IN]) -> Iterable[OUT]:
Review comment:
This's an alignment to the java code, which is used to make a generic difference between single-value functions and iterable functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19054:
URL: https://github.com/apache/flink/pull/19054#issuecomment-1064837015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32889",
"triggerID" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"triggerType" : "PUSH"
}, {
"hash" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32989",
"triggerID" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"triggerType" : "PUSH"
}, {
"hash" : "591c73b1c352667583b076abb19e8f259fbb4d38",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33140",
"triggerID" : "591c73b1c352667583b076abb19e8f259fbb4d38",
"triggerType" : "PUSH"
}, {
"hash" : "d58e60288dc587f40e57565a76973488ea692ad2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d58e60288dc587f40e57565a76973488ea692ad2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 591c73b1c352667583b076abb19e8f259fbb4d38 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33140)
* d58e60288dc587f40e57565a76973488ea692ad2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19054:
URL: https://github.com/apache/flink/pull/19054#issuecomment-1064837015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32889",
"triggerID" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"triggerType" : "PUSH"
}, {
"hash" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32989",
"triggerID" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"triggerType" : "PUSH"
}, {
"hash" : "591c73b1c352667583b076abb19e8f259fbb4d38",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33140",
"triggerID" : "591c73b1c352667583b076abb19e8f259fbb4d38",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 591c73b1c352667583b076abb19e8f259fbb4d38 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=33140)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #19054: [FLINK-26482][python] Support WindowedStream.reduce in Python DataStream API
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #19054:
URL: https://github.com/apache/flink/pull/19054#issuecomment-1064837015
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32889",
"triggerID" : "5c14b7fc09537f4959b8dccc25722140ad2e24ba",
"triggerType" : "PUSH"
}, {
"hash" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32989",
"triggerID" : "7d95b4714a85b0435ef78757e200c2ba92ee22e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5c14b7fc09537f4959b8dccc25722140ad2e24ba Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32889)
* 7d95b4714a85b0435ef78757e200c2ba92ee22e8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32989)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org