You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/12/04 15:59:18 UTC
[GitHub] [flink] aljoscha opened a new pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
aljoscha opened a new pull request #14312:
URL: https://github.com/apache/flink/pull/14312
Right now, we don't support `DataStream.connect(BroadcastStream)` in `BATCH` execution mode. I believe we can add support for this with not too much work.
The key insight is that we can process the broadcast side before the non-broadcast side. Initially, we were shying away from this because of concerns about `ctx.applyToKeyedState()` which allows the broadcast side of the user function to access/iterate over state from the keyed side. We thought that we couldn't support this. However, since we know that we process the broadcast side first we know that the keyed side will always be empty when doing so. We can thus just make this "keyed iteration" call a no-op, instead of throwing an exception as we do now.
This is work-in-progress. I have yet to add tests.
## Brief change log
* turn broadcast operation into a logical `Transformation`
* add `BATCH` operators for broadcast
* add translators
## Verifying this change
To be done.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? maybe
We need to remove the caveat from the documentation. 🎊
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49ca15835522bee396ea7dc88c1ab060e14bd7cb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599)
* bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
}, {
"hash" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643",
"triggerID" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"triggerType" : "PUSH"
}, {
"hash" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11662",
"triggerID" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"triggerType" : "PUSH"
}, {
"hash" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11665",
"triggerID" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"triggerType" : "PUSH"
}, {
"hash" : "f0698650b76335ee6a8a9b3079e70e853040da59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11691",
"triggerID" : "f0698650b76335ee6a8a9b3079e70e853040da59",
"triggerType" : "PUSH"
}, {
"hash" : "ca72d305e20fbb2ab11a36b93a2a1e969b6b7883",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11703",
"triggerID" : "ca72d305e20fbb2ab11a36b93a2a1e969b6b7883",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca72d305e20fbb2ab11a36b93a2a1e969b6b7883 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11703)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f76d4c8d68a126aaf9a52b2deb99500a76ed412d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763)
* 6f0b8a4de3ba32b1cf407287add34dad903a68a2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551355827
##########
File path: flink-tests/src/test/java/org/apache/flink/api/datastream/DataStreamBatchExecutionITCase.java
##########
@@ -180,6 +191,107 @@ public void batchSumSingleResultPerKey() throws Exception {
}
}
+ /** Verifies that all broadcast input is processed before keyed input. */
+ @Test
+ public void batchKeyedBroadcastExecution() throws Exception {
Review comment:
adding it to the test 👌
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
}, {
"hash" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643",
"triggerID" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"triggerType" : "PUSH"
}, {
"hash" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11662",
"triggerID" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"triggerType" : "PUSH"
}, {
"hash" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11665",
"triggerID" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"triggerType" : "PUSH"
}, {
"hash" : "f0698650b76335ee6a8a9b3079e70e853040da59",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f0698650b76335ee6a8a9b3079e70e853040da59",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 30f17119c79e6800df9800df9cf63a700f7e2624 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11665)
* f0698650b76335ee6a8a9b3079e70e853040da59 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551461770
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamMultipleInputProcessorFactory.java
##########
@@ -142,6 +148,7 @@ public static StreamMultipleInputProcessor create(
idx, userClassloader))
.toArray(TypeSerializer[]::new),
streamConfig.getStateKeySerializer(userClassloader),
+ new StreamTaskInput[0],
Review comment:
I added support for mixed two-input operations, along with tests. I found some bugs along the way, the fixes are individual commits to make it easier to follow/review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7d1707557148278604a497b07e2682b342da246 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863)
* 1e4d45934795e74502b7871d1cbad7380727ad22 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023)
* 091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r541083223
##########
File path: docs/dev/datastream_execution_mode.md
##########
@@ -237,6 +237,36 @@ next key.
See [FLIP-140](https://cwiki.apache.org/confluence/x/kDh4CQ) for background
information on this.
+### Order of Processing
+
+The order in which records are processed in operators or user defined functions
+(UDFs) can differ between `BATCH` and `STREAMING` execution.
+
+In `STREAMING` mode, user defined functions should not make any assumptions
+about the order of incoming records. Records are processed immediately once
+they arrive in sources.
+
+In `BATCH` execution mode, there are some operations where Flink guarantees
+order. The ordering can be a side effect of the special task scheduling,
+network shuffle, and state backend (see above) or it can be a conscious choice
+by the system.
+
+There are three general types of input that we can differentiate:
+
+- _keyed input_: input from a `KeyedStream`
+- _broadcast input_: input from a broadcast stream (see also [Broadcast
+ State]({% link dev/stream/state/broadcast_state.md %}))
+- _regular input_: input that isn't any of the above types of input
+
+These are the ordering rules for the different input types
+
+- keyed inputs are processed after all other inputs
+- broadcast inputs are processed before regular inputs
+
+As mentioned above, the keyed input will be grouped and Flink will process all
+records of a keyed group consecutively before processing the next group.
Review comment:
@aljoscha What about this?
```suggestion
### Order of Processing
The order in which records are processed in operators or user-defined functions (UDFs) can differ between `BATCH` and `STREAMING` execution.
In `STREAMING` mode, user-defined functions should not make any assumptions about incoming records' order.
Data is processed as soon as it arrives.
In `BATCH` execution mode, there are some operations where Flink guarantees order.
The ordering can be a side effect of the particular task scheduling,
network shuffle, and state backend (see above), or a conscious choice by the system.
There are three general types of input that we can differentiate:
- _broadcast input_: input from a broadcast stream (see also [Broadcast
State]({% link dev/stream/state/broadcast_state.md %}))
- _regular input_: input that isn't any of the above types of input
- _keyed input_: input from a `KeyedStream`
Functions, or Operators, that consume multiple input types will process them in the following order:
- broadcast inputs are processed first
- regular inputs are processed second
- keyed inputs are processed last
For functions that consume from multiple regular or broadcast inputs — such as a `CoProcessFunction` — Flink has the right to process data from any input of that type in any order.
For functions that consume from multiple keyed inputs — such as a `KeyedCoProcessFunction` — Flink processes all records for a single key from all keyed inputs before moving on to the next.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
}, {
"hash" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643",
"triggerID" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"triggerType" : "PUSH"
}, {
"hash" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643)
* 84d2f162af19db4ca01d2015e81d4e71533c9a01 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7d1707557148278604a497b07e2682b342da246 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863)
* 1e4d45934795e74502b7871d1cbad7380727ad22 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808)
* 9f3deb69af6892fc676aa87e1d1b55981eded305 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 73df4e8f714ac2ae08213542130d91196dc50792 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-756055834
Merged this manually
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551357346
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/TwoInputTransformationTranslator.java
##########
@@ -46,7 +47,11 @@
transformation.getStateKeySelector1() != null
&& transformation.getStateKeySelector2() != null;
if (isKeyed) {
Review comment:
I think we have to yes. I shied away from it here because I would have to add a test that uses the low-level API because right now it's not possible for users to create such transforms.
Could you maybe whip together a test based on the existing test code you already had? We add it here and I change it to correctly forward the requirements?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 73df4e8f714ac2ae08213542130d91196dc50792 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545)
* 37d3b02971986addded4356b88a493b824244b2e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551790684
##########
File path: docs/dev/datastream_execution_mode.md
##########
@@ -1,8 +1,6 @@
---
title: "Execution Mode (Batch/Streaming)"
-nav-id: datastream_execution_mode
-nav-parent_id: streaming
-nav-pos: 1
+nav-id: datastream_execution_mode nav-parent_id: streaming nav-pos: 1
Review comment:
Are all the changes in the file required? Moreover I think this change to the header may break the page.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r540218683
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/TwoInputTransformationTranslator.java
##########
@@ -45,7 +46,11 @@
boolean isKeyed =
transformation.getStateKeySelector1() != null && transformation.getStateKeySelector2() != null;
if (isKeyed) {
Review comment:
I went with the current behaviour, but yes, we could do that. So in that case the non-keyed input would be processed before all the keyed input. Same for the general multi-input.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1e4d45934795e74502b7871d1cbad7380727ad22 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023)
* 091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f3deb69af6892fc676aa87e1d1b55981eded305 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814)
* 6d0f9c6a32d65c82e657cfcab048a98aa4445429 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073)
* 8dd56a728ce8bf762ccd69d2d9131de62319064f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551353891
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamTwoInputProcessorFactory.java
##########
@@ -90,24 +94,51 @@
InputSelectable inputSelectable =
streamOperator instanceof InputSelectable ? (InputSelectable) streamOperator : null;
- if (streamConfig.shouldSortInputs()) {
+
+ // this is a bit verbose because we're manually handling input1 and input2
+ // TODO: extract method
+ StreamConfig.InputConfig[] inputConfigs = streamConfig.getInputs(userClassloader);
+ boolean input1IsSorted = requiresSorting(inputConfigs[0]);
+ boolean input2IsSorted = requiresSorting(inputConfigs[1]);
+
+ if (input1IsSorted || input2IsSorted) {
+ // as soon as one input requires sorting we need to treat all inputs differently, to
+ // make sure that pass-through inputs have precedence
if (inputSelectable != null) {
throw new IllegalStateException(
"The InputSelectable interface is not supported with sorting inputs");
}
- @SuppressWarnings("unchecked")
+ List<StreamTaskInput<?>> sortedTaskInputs = new ArrayList<>();
+ List<StreamTaskInput<?>> passThroughTaskInputs = new ArrayList<>();
+ int input1Index = 0;
+ int input2Index;
+ if (input1IsSorted) {
+ sortedTaskInputs.add(input1);
+ } else {
+ passThroughTaskInputs.add(input1);
+ }
+ if (input2IsSorted) {
+ input2Index = sortedTaskInputs.size();
+ sortedTaskInputs.add(input2);
+ } else {
+ input2Index = passThroughTaskInputs.size();
+ passThroughTaskInputs.add(input2);
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
}, {
"hash" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643",
"triggerID" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"triggerType" : "PUSH"
}, {
"hash" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11662",
"triggerID" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643)
* 84d2f162af19db4ca01d2015e81d4e71533c9a01 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11662)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f3deb69af6892fc676aa87e1d1b55981eded305 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814)
* 6d0f9c6a32d65c82e657cfcab048a98aa4445429 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849)
* 897fe1eb2bb46f63f8cea373c5b5d917070823b5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851)
* b7d1707557148278604a497b07e2682b342da246 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551461439
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/MultiInputSortingDataInput.java
##########
@@ -107,29 +112,38 @@ private MultiInputSortingDataInput(
*/
public static class SelectableSortingInputs {
Review comment:
results are now on http://codespeed.dak8s.net:8000/comparison/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551351615
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamMultipleInputProcessorFactory.java
##########
@@ -142,6 +148,7 @@ public static StreamMultipleInputProcessor create(
idx, userClassloader))
.toArray(TypeSerializer[]::new),
streamConfig.getStateKeySerializer(userClassloader),
+ new StreamTaskInput[0],
Review comment:
It's technically wrong but we don't allow mixed sorted/non-sorted inputs right now, I think.
In general, the support for mixed inputs is in flux with this PR, also relevant for https://github.com/apache/flink/pull/14324.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aaa5bc44d948629278738fd441d711b32c877fba Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551228929
##########
File path: docs/dev/datastream_execution_mode.md
##########
@@ -237,6 +237,35 @@ next key.
See [FLIP-140](https://cwiki.apache.org/confluence/x/kDh4CQ) for background
information on this.
+### Order of Processing
+
+The order in which records are processed in operators or user-defined functions (UDFs) can differ between `BATCH` and `STREAMING` execution.
+
+In `STREAMING` mode, user-defined functions should not make any assumptions about incoming records' order.
+Data is processed as soon as it arrives.
+
+In `BATCH` execution mode, there are some operations where Flink guarantees order.
+The ordering can be a side effect of the particular task scheduling,
+network shuffle, and state backend (see above), or a conscious choice by the system.
+
+There are three general types of input that we can differentiate:
+
+- _broadcast input_: input from a broadcast stream (see also [Broadcast
+ State]({% link dev/stream/state/broadcast_state.md %}))
+- _regular input_: input that isn't any of the above types of input
Review comment:
shouldn't it be below the keyed input?
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/MultiInputSortingDataInput.java
##########
@@ -318,23 +346,40 @@ private InputStatus addNextToQueue(HeadElement reuse, DataOutput<IN> output) thr
* all sorting inputs. Should be used by the {@link StreamInputProcessor} to choose the next
* input to consume from.
*/
- private static class InputSelector implements InputSelectable {
+ private static class InputSelector implements InputSelectable, BoundedMultiInput {
private final CommonContext commonContext;
- private final int numberOfInputs;
+ private final int numInputs;
+ private final ConcurrentLinkedQueue<Integer> passThroughInputsIndices;
Review comment:
I'd rather not use the concurrent collection. Unless you are sure it is required. If so I'd put a comment why is it so.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/MultiInputSortingDataInput.java
##########
@@ -107,29 +112,38 @@ private MultiInputSortingDataInput(
*/
public static class SelectableSortingInputs {
Review comment:
I think it would make sense to run the benchmarks over the change to see if we have a performance penalty, especially in the https://github.com/apache/flink-benchmarks/blob/0843ba6c8e2e8c856d4da333e546c3077220c7c8/src/main/java/org/apache/flink/benchmark/SortingBoundedInputBenchmarks.java#L157
(there is some instruction how to run the benchmarks)
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847
##########
File path: flink-tests/src/test/java/org/apache/flink/api/datastream/DataStreamBatchExecutionITCase.java
##########
@@ -180,6 +191,107 @@ public void batchSumSingleResultPerKey() throws Exception {
}
}
+ /** Verifies that all broadcast input is processed before keyed input. */
+ @Test
+ public void batchKeyedBroadcastExecution() throws Exception {
Review comment:
nit: How about we test with at least two keys?
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamMultipleInputProcessorFactory.java
##########
@@ -142,6 +148,7 @@ public static StreamMultipleInputProcessor create(
idx, userClassloader))
.toArray(TypeSerializer[]::new),
streamConfig.getStateKeySerializer(userClassloader),
+ new StreamTaskInput[0],
Review comment:
This is wrong, right?
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamTwoInputProcessorFactory.java
##########
@@ -90,24 +94,51 @@
InputSelectable inputSelectable =
streamOperator instanceof InputSelectable ? (InputSelectable) streamOperator : null;
- if (streamConfig.shouldSortInputs()) {
+
+ // this is a bit verbose because we're manually handling input1 and input2
+ // TODO: extract method
+ StreamConfig.InputConfig[] inputConfigs = streamConfig.getInputs(userClassloader);
+ boolean input1IsSorted = requiresSorting(inputConfigs[0]);
+ boolean input2IsSorted = requiresSorting(inputConfigs[1]);
+
+ if (input1IsSorted || input2IsSorted) {
+ // as soon as one input requires sorting we need to treat all inputs differently, to
+ // make sure that pass-through inputs have precedence
if (inputSelectable != null) {
throw new IllegalStateException(
"The InputSelectable interface is not supported with sorting inputs");
}
- @SuppressWarnings("unchecked")
+ List<StreamTaskInput<?>> sortedTaskInputs = new ArrayList<>();
+ List<StreamTaskInput<?>> passThroughTaskInputs = new ArrayList<>();
+ int input1Index = 0;
+ int input2Index;
+ if (input1IsSorted) {
+ sortedTaskInputs.add(input1);
+ } else {
+ passThroughTaskInputs.add(input1);
+ }
+ if (input2IsSorted) {
+ input2Index = sortedTaskInputs.size();
+ sortedTaskInputs.add(input2);
+ } else {
+ input2Index = passThroughTaskInputs.size();
+ passThroughTaskInputs.add(input2);
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
Review comment:
nit: my IDE complains the `"rawtypes"` is unnecessary.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/AbstractBroadcastStateTransformation.java
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Base class for Broadcast State transformations. In a nutshell, this transformation allows to take
+ * a broadcast (non-keyed) stream, connect it with another keyed or non-keyed stream, and apply a
+ * function on the resulting connected stream. This function will have access to all the elements
+ * that belong to the non-keyed, broadcasted side, as this is kept in Flink's state.
Review comment:
nit: Is the sentence correct? I think it will have access to the state populated with the elements of the broadcast side. Not necessarily to all the elements or elements in general.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/BroadcastConnectedStream.java
##########
@@ -246,28 +246,36 @@ public StreamExecutionEnvironment getExecutionEnvironment() {
return returnStream;
}
- private <OUT> BroadcastStateTransformation<IN1, IN2, OUT> getBroadcastStateTransformation(
+ @Internal
+ private <KEY, OUT> SingleOutputStreamOperator<OUT> transform(
final String functionName,
- final TypeInformation<OUT> outTypeInfo,
- final TwoInputStreamOperator<IN1, IN2, OUT> operator) {
-
- if (nonBroadcastStream instanceof KeyedStream) {
- return BroadcastStateTransformation.forKeyedStream(
- functionName,
- (KeyedStream<IN1, ?>) nonBroadcastStream,
- broadcastStream,
- SimpleOperatorFactory.of(operator),
- outTypeInfo,
- environment.getParallelism());
- } else {
- return BroadcastStateTransformation.forNonKeyedStream(
- functionName,
- nonBroadcastStream,
- broadcastStream,
- SimpleOperatorFactory.of(operator),
- outTypeInfo,
- environment.getParallelism());
- }
+ final KeyedBroadcastProcessFunction<KEY, IN1, IN2, OUT> userFunction,
+ final TypeInformation<OUT> outTypeInfo) {
+
+ // read the output type of the input Transforms to coax out errors about MissingTypeInfo
+ nonBroadcastStream.getType();
+ broadcastStream.getType();
+
+ KeyedStream<IN1, KEY> keyedInputStream = (KeyedStream<IN1, KEY>) nonBroadcastStream;
+
+ final KeyedBroadcastStateTransformation<KEY, IN1, IN2, OUT> transformation =
+ new KeyedBroadcastStateTransformation<>(
+ functionName,
Review comment:
nit: How about we inline the functionName(s) in both `transform` methods.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/TwoInputTransformationTranslator.java
##########
@@ -46,7 +47,11 @@
transformation.getStateKeySelector1() != null
&& transformation.getStateKeySelector2() != null;
if (isKeyed) {
Review comment:
I think we should change it so that we apply sorting if any of the inputs is keyed.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamTwoInputProcessorFactory.java
##########
@@ -90,24 +94,51 @@
InputSelectable inputSelectable =
streamOperator instanceof InputSelectable ? (InputSelectable) streamOperator : null;
- if (streamConfig.shouldSortInputs()) {
+
+ // this is a bit verbose because we're manually handling input1 and input2
+ // TODO: extract method
+ StreamConfig.InputConfig[] inputConfigs = streamConfig.getInputs(userClassloader);
+ boolean input1IsSorted = requiresSorting(inputConfigs[0]);
+ boolean input2IsSorted = requiresSorting(inputConfigs[1]);
+
+ if (input1IsSorted || input2IsSorted) {
+ // as soon as one input requires sorting we need to treat all inputs differently, to
+ // make sure that pass-through inputs have precedence
if (inputSelectable != null) {
throw new IllegalStateException(
"The InputSelectable interface is not supported with sorting inputs");
}
- @SuppressWarnings("unchecked")
+ List<StreamTaskInput<?>> sortedTaskInputs = new ArrayList<>();
+ List<StreamTaskInput<?>> passThroughTaskInputs = new ArrayList<>();
+ int input1Index = 0;
Review comment:
It is rather a matter of preference, but I'd move the index computing down below, where it is needed. I found it a bit confusing here and the IDE complains that in line 126 the `passThroughTaskInputs.size()` is always `0`.
We won't need this logic here, if we change the lines 151-160 to:
```StreamTaskInput<?>[] sortedInputs = selectableSortingInputs.getSortedInputs();
StreamTaskInput<?>[] passThroughInputs = selectableSortingInputs.getPassThroughInputs();
if (input1IsSorted) {
input1 = toTypedInput(sortedInputs[0]);
} else {
input1 = toTypedInput(passThroughInputs[0]);
}
if (input2IsSorted) {
input2 = toTypedInput(sortedInputs[sortedInputs.length - 1]);
} else {
input2 = toTypedInput(passThroughInputs[passThroughInputs.length - 1]);
}
```
We can base that on the assumption that the `input1` will be first in the corresponding group while the `input2` last.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738864641
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit ca72d305e20fbb2ab11a36b93a2a1e969b6b7883 (Fri May 28 07:01:48 UTC 2021)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551359341
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/MultiInputSortingDataInput.java
##########
@@ -318,23 +346,40 @@ private InputStatus addNextToQueue(HeadElement reuse, DataOutput<IN> output) thr
* all sorting inputs. Should be used by the {@link StreamInputProcessor} to choose the next
* input to consume from.
*/
- private static class InputSelector implements InputSelectable {
+ private static class InputSelector implements InputSelectable, BoundedMultiInput {
private final CommonContext commonContext;
- private final int numberOfInputs;
+ private final int numInputs;
+ private final ConcurrentLinkedQueue<Integer> passThroughInputsIndices;
Review comment:
changing to a `LinkedList` 👌
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57afbcf067fb64d6e669a45accbb95aaca082345 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-742390910
I am good with the general idea and how it looks inside the `MultiInputSortingDataInput.java`. Unless someone objects/has a different opinion, I think we can start properly wiring all the inputs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-754499351
What is the plan for mixed keyed/non-keyed inputs in multiple inputs? If I am not mistaken the implementation in the `StreamMultipleInputProcessorFactory` is not complete and it does not set the pass through inputs correctly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r545741434
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamNode.java
##########
@@ -84,7 +84,7 @@
private String transformationUID;
private String userHash;
- private boolean sortedInputs = false;
+ private Map<Integer, StreamConfig.InputRequirement> inputRequirements = new HashMap<>();
Review comment:
None of the others are initialized in the constructor that's why I went with this one. But yes, I would agree!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r546347104
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamMultipleInputProcessorFactory.java
##########
@@ -143,7 +150,20 @@ else if (configuredInput instanceof StreamConfig.SourceInputConfig) {
jobConfig
);
- inputs = selectableSortingInputs.getSortingInputs();
+ StreamTaskInput<?>[] sortedInputs = selectableSortingInputs.getSortedInputs();
+ StreamTaskInput<?>[] passThroughInputs = selectableSortingInputs.getPassThroughInputs();
+ int sortedIndex = 0;
+ int passThroughIndex = 0;
+ for (int i = 0; i < inputs.length; i++) {
+ if (requiresSorting(inputConfigs[i])) {
+ inputs[i] = sortedInputs[sortedIndex];
+ sortedIndex++;
+ } else {
+ inputs[i] = passThroughInputs[passThroughIndex];
+ passThroughIndex++;
+ }
+ }
+ inputs = selectableSortingInputs.getSortedInputs();
Review comment:
Yes, it's a leftover. 🙈
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551350742
##########
File path: docs/dev/datastream_execution_mode.md
##########
@@ -237,6 +237,35 @@ next key.
See [FLIP-140](https://cwiki.apache.org/confluence/x/kDh4CQ) for background
information on this.
+### Order of Processing
+
+The order in which records are processed in operators or user-defined functions (UDFs) can differ between `BATCH` and `STREAMING` execution.
+
+In `STREAMING` mode, user-defined functions should not make any assumptions about incoming records' order.
+Data is processed as soon as it arrives.
+
+In `BATCH` execution mode, there are some operations where Flink guarantees order.
+The ordering can be a side effect of the particular task scheduling,
+network shuffle, and state backend (see above), or a conscious choice by the system.
+
+There are three general types of input that we can differentiate:
+
+- _broadcast input_: input from a broadcast stream (see also [Broadcast
+ State]({% link dev/stream/state/broadcast_state.md %}))
+- _regular input_: input that isn't any of the above types of input
Review comment:
I think I might've not been clear on this one. What I meant is that the line with `regular input` should be below the `keyed input`, because it refers to it. If I understood the sentence correctly it says: `input that isn't broadcast or keyed`, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551352025
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/AbstractBroadcastStateTransformation.java
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Base class for Broadcast State transformations. In a nutshell, this transformation allows to take
+ * a broadcast (non-keyed) stream, connect it with another keyed or non-keyed stream, and apply a
+ * function on the resulting connected stream. This function will have access to all the elements
+ * that belong to the non-keyed, broadcasted side, as this is kept in Flink's state.
Review comment:
I'd be ok with simply dropping it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2bc9331d2652a2e47038f885036b01ea936e48f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
}, {
"hash" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643",
"triggerID" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"triggerType" : "PUSH"
}, {
"hash" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11662",
"triggerID" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"triggerType" : "PUSH"
}, {
"hash" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11665",
"triggerID" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"triggerType" : "PUSH"
}, {
"hash" : "f0698650b76335ee6a8a9b3079e70e853040da59",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11691",
"triggerID" : "f0698650b76335ee6a8a9b3079e70e853040da59",
"triggerType" : "PUSH"
}, {
"hash" : "ca72d305e20fbb2ab11a36b93a2a1e969b6b7883",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11703",
"triggerID" : "ca72d305e20fbb2ab11a36b93a2a1e969b6b7883",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0698650b76335ee6a8a9b3079e70e853040da59 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11691)
* ca72d305e20fbb2ab11a36b93a2a1e969b6b7883 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11703)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r540208666
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/MultiInputTransformationTranslator.java
##########
@@ -54,7 +55,11 @@
Collection<Integer> ids = translateInternal(transformation, context);
boolean isKeyed = transformation instanceof KeyedMultipleInputTransformation;
Review comment:
I think this check is not sufficient now. We should check if any of the inputs has a state selector.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f3deb69af6892fc676aa87e1d1b55981eded305 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814)
* 6d0f9c6a32d65c82e657cfcab048a98aa4445429 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849)
* 897fe1eb2bb46f63f8cea373c5b5d917070823b5 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551345782
##########
File path: docs/dev/datastream_execution_mode.md
##########
@@ -237,6 +237,35 @@ next key.
See [FLIP-140](https://cwiki.apache.org/confluence/x/kDh4CQ) for background
information on this.
+### Order of Processing
+
+The order in which records are processed in operators or user-defined functions (UDFs) can differ between `BATCH` and `STREAMING` execution.
+
+In `STREAMING` mode, user-defined functions should not make any assumptions about incoming records' order.
+Data is processed as soon as it arrives.
+
+In `BATCH` execution mode, there are some operations where Flink guarantees order.
+The ordering can be a side effect of the particular task scheduling,
+network shuffle, and state backend (see above), or a conscious choice by the system.
+
+There are three general types of input that we can differentiate:
+
+- _broadcast input_: input from a broadcast stream (see also [Broadcast
+ State]({% link dev/stream/state/broadcast_state.md %}))
+- _regular input_: input that isn't any of the above types of input
Review comment:
regular input should be processed before keyed input, because the (sorted) keyed input is always last.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49ca15835522bee396ea7dc88c1ab060e14bd7cb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551308750
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/TwoInputTransformationTranslator.java
##########
@@ -46,7 +47,11 @@
transformation.getStateKeySelector1() != null
&& transformation.getStateKeySelector2() != null;
if (isKeyed) {
Review comment:
I think we should change it so that we apply sorting if any of the inputs is keyed. Otherwise the combination with the BATCH state backend will lead to wrong results.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-743264798
@sjwiesman I'm also adding documentation here, it's a bit terse but still helpful, I think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r545737859
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java
##########
@@ -735,4 +761,10 @@ public int hashCode() {
return inputEdge.hashCode();
}
}
+
Review comment:
I decided to put it into the `NetworkInputConfig` because that's the only place where it can really be applied right now. But I can also move it to the base class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
}, {
"hash" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643",
"triggerID" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"triggerType" : "PUSH"
}, {
"hash" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11662",
"triggerID" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"triggerType" : "PUSH"
}, {
"hash" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11665",
"triggerID" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 84d2f162af19db4ca01d2015e81d4e71533c9a01 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11662)
* 30f17119c79e6800df9800df9cf63a700f7e2624 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11665)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
}, {
"hash" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624)
* 17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551300493
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamMultipleInputProcessorFactory.java
##########
@@ -142,6 +148,7 @@ public static StreamMultipleInputProcessor create(
idx, userClassloader))
.toArray(TypeSerializer[]::new),
streamConfig.getStateKeySerializer(userClassloader),
+ new StreamTaskInput[0],
Review comment:
This is wrong, right? We should extract the pass through inputs from the config, shouldn't we?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042)
* a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r540289201
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/TwoInputTransformationTranslator.java
##########
@@ -45,7 +46,11 @@
boolean isKeyed =
transformation.getStateKeySelector1() != null && transformation.getStateKeySelector2() != null;
if (isKeyed) {
Review comment:
I'll do that then. 👌
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r540209912
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/TwoInputTransformationTranslator.java
##########
@@ -45,7 +46,11 @@
boolean isKeyed =
transformation.getStateKeySelector1() != null && transformation.getStateKeySelector2() != null;
if (isKeyed) {
Review comment:
Similar to the `MultiInput` shall we account for the situation if only one input is keyed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r539998701
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/MultiInputSortingDataInput.java
##########
@@ -315,23 +342,40 @@ private InputStatus addNextToQueue(HeadElement reuse, DataOutput<IN> output) thr
* element of all sorting inputs. Should be used by the {@link StreamInputProcessor} to
* choose the next input to consume from.
*/
- private static class InputSelector implements InputSelectable {
+ private static class InputSelector implements InputSelectable, BoundedMultiInput {
private final CommonContext commonContext;
- private final int numberOfInputs;
+ private final int numInputs;
+ private final ConcurrentLinkedQueue<Integer> passThroughInputsIndices;
Review comment:
Do we need the `Concurrent` queue here? If I am not mistaken all of it happens from within the main mailbox thread.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
sjwiesman commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r541083223
##########
File path: docs/dev/datastream_execution_mode.md
##########
@@ -237,6 +237,36 @@ next key.
See [FLIP-140](https://cwiki.apache.org/confluence/x/kDh4CQ) for background
information on this.
+### Order of Processing
+
+The order in which records are processed in operators or user defined functions
+(UDFs) can differ between `BATCH` and `STREAMING` execution.
+
+In `STREAMING` mode, user defined functions should not make any assumptions
+about the order of incoming records. Records are processed immediately once
+they arrive in sources.
+
+In `BATCH` execution mode, there are some operations where Flink guarantees
+order. The ordering can be a side effect of the special task scheduling,
+network shuffle, and state backend (see above) or it can be a conscious choice
+by the system.
+
+There are three general types of input that we can differentiate:
+
+- _keyed input_: input from a `KeyedStream`
+- _broadcast input_: input from a broadcast stream (see also [Broadcast
+ State]({% link dev/stream/state/broadcast_state.md %}))
+- _regular input_: input that isn't any of the above types of input
+
+These are the ordering rules for the different input types
+
+- keyed inputs are processed after all other inputs
+- broadcast inputs are processed before regular inputs
+
+As mentioned above, the keyed input will be grouped and Flink will process all
+records of a keyed group consecutively before processing the next group.
Review comment:
@aljoscha What about this?
```suggestion
### Order of Processing
The order in which records are processed in operators or user-defined functions (UDFs) can differ between `BATCH` and `STREAMING` execution.
In `STREAMING` mode, user-defined functions should not make any assumptions about incoming records' order.
Data is processed as soon as it arrives.
In `BATCH` execution mode, there are some operations where Flink guarantees order.
The ordering can be a side effect of the particular task scheduling,
network shuffle, and state backend (see above), or a conscious choice by the system.
There are three general types of input that we can differentiate:
- _broadcast input_: input from a broadcast stream (see also [Broadcast
State]({% link dev/stream/state/broadcast_state.md %}))
- _regular input_: input that isn't any of the above types of input
- _keyed input_: input from a `KeyedStream`
Functions, or Operators, that consume multiple input types will process them in the following order:
- broadcast inputs are processed first
- regular inputs are processed second
- keyed inputs are processed last
For functions that consume from multiple regular or broadcast inputs — such as a `CoProcessFunction` — Flink has the right to process data from any input of that type in any order.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
}, {
"hash" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643",
"triggerID" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"triggerType" : "PUSH"
}, {
"hash" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11662",
"triggerID" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"triggerType" : "PUSH"
}, {
"hash" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11665",
"triggerID" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"triggerType" : "PUSH"
}, {
"hash" : "f0698650b76335ee6a8a9b3079e70e853040da59",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11691",
"triggerID" : "f0698650b76335ee6a8a9b3079e70e853040da59",
"triggerType" : "PUSH"
}, {
"hash" : "ca72d305e20fbb2ab11a36b93a2a1e969b6b7883",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ca72d305e20fbb2ab11a36b93a2a1e969b6b7883",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0698650b76335ee6a8a9b3079e70e853040da59 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11691)
* ca72d305e20fbb2ab11a36b93a2a1e969b6b7883 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551824176
##########
File path: docs/dev/datastream_execution_mode.md
##########
@@ -1,8 +1,6 @@
---
title: "Execution Mode (Batch/Streaming)"
-nav-id: datastream_execution_mode
-nav-parent_id: streaming
-nav-pos: 1
+nav-id: datastream_execution_mode nav-parent_id: streaming nav-pos: 1
Review comment:
I don't know how those snuck in... 🙈, reverting
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551349489
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/AbstractBroadcastStateTransformation.java
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Base class for Broadcast State transformations. In a nutshell, this transformation allows to take
+ * a broadcast (non-keyed) stream, connect it with another keyed or non-keyed stream, and apply a
+ * function on the resulting connected stream. This function will have access to all the elements
+ * that belong to the non-keyed, broadcasted side, as this is kept in Flink's state.
Review comment:
It's not correct and I copied it from the previous transformation. I can remove the sentence or try and fix it. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042)
* a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] kl0u commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
kl0u commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r546289324
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamMultipleInputProcessorFactory.java
##########
@@ -143,7 +150,20 @@ else if (configuredInput instanceof StreamConfig.SourceInputConfig) {
jobConfig
);
- inputs = selectableSortingInputs.getSortingInputs();
+ StreamTaskInput<?>[] sortedInputs = selectableSortingInputs.getSortedInputs();
+ StreamTaskInput<?>[] passThroughInputs = selectableSortingInputs.getPassThroughInputs();
+ int sortedIndex = 0;
+ int passThroughIndex = 0;
+ for (int i = 0; i < inputs.length; i++) {
+ if (requiresSorting(inputConfigs[i])) {
+ inputs[i] = sortedInputs[sortedIndex];
+ sortedIndex++;
+ } else {
+ inputs[i] = passThroughInputs[passThroughIndex];
+ passThroughIndex++;
+ }
+ }
+ inputs = selectableSortingInputs.getSortedInputs();
Review comment:
Seems like a leftover line but that can lead to bugs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f3deb69af6892fc676aa87e1d1b55981eded305 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
}, {
"hash" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643",
"triggerID" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"triggerType" : "PUSH"
}, {
"hash" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11662",
"triggerID" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"triggerType" : "PUSH"
}, {
"hash" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643)
* 84d2f162af19db4ca01d2015e81d4e71533c9a01 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11662)
* 30f17119c79e6800df9800df9cf63a700f7e2624 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738864641
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 57afbcf067fb64d6e669a45accbb95aaca082345 (Fri Dec 04 16:01:18 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aaa5bc44d948629278738fd441d711b32c877fba Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542)
* 73df4e8f714ac2ae08213542130d91196dc50792 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551366952
##########
File path: docs/dev/datastream_execution_mode.md
##########
@@ -237,6 +237,35 @@ next key.
See [FLIP-140](https://cwiki.apache.org/confluence/x/kDh4CQ) for background
information on this.
+### Order of Processing
+
+The order in which records are processed in operators or user-defined functions (UDFs) can differ between `BATCH` and `STREAMING` execution.
+
+In `STREAMING` mode, user-defined functions should not make any assumptions about incoming records' order.
+Data is processed as soon as it arrives.
+
+In `BATCH` execution mode, there are some operations where Flink guarantees order.
+The ordering can be a side effect of the particular task scheduling,
+network shuffle, and state backend (see above), or a conscious choice by the system.
+
+There are three general types of input that we can differentiate:
+
+- _broadcast input_: input from a broadcast stream (see also [Broadcast
+ State]({% link dev/stream/state/broadcast_state.md %}))
+- _regular input_: input that isn't any of the above types of input
Review comment:
Ah damn, you're right... 🙈 fixing it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808)
* 9f3deb69af6892fc676aa87e1d1b55981eded305 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 37d3b02971986addded4356b88a493b824244b2e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602)
* d2bc9331d2652a2e47038f885036b01ea936e48f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57afbcf067fb64d6e669a45accbb95aaca082345 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f76d4c8d68a126aaf9a52b2deb99500a76ed412d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r540228829
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/TwoInputTransformationTranslator.java
##########
@@ -45,7 +46,11 @@
boolean isKeyed =
transformation.getStateKeySelector1() != null && transformation.getStateKeySelector2() != null;
if (isKeyed) {
Review comment:
Yes, I'd do it that way. Otherwise I'd proceed with https://github.com/apache/flink/pull/14324. The current state leads to possibly corrupted results.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6d0f9c6a32d65c82e657cfcab048a98aa4445429 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849)
* 897fe1eb2bb46f63f8cea373c5b5d917070823b5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851)
* b7d1707557148278604a497b07e2682b342da246 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57afbcf067fb64d6e669a45accbb95aaca082345 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534)
* aaa5bc44d948629278738fd441d711b32c877fba UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-742614207
I updated this with tests and more Javadocs. PTAL.
I still have to add support for multi-input operators and I need to enable mixed keyed/non-keyed operators, as per Dawid's suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551461102
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/AbstractBroadcastStateTransformation.java
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Base class for Broadcast State transformations. In a nutshell, this transformation allows to take
+ * a broadcast (non-keyed) stream, connect it with another keyed or non-keyed stream, and apply a
+ * function on the resulting connected stream. This function will have access to all the elements
+ * that belong to the non-keyed, broadcasted side, as this is kept in Flink's state.
Review comment:
dropping it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 49ca15835522bee396ea7dc88c1ab060e14bd7cb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599)
* bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551950784
##########
File path: flink-tests/src/test/java/org/apache/flink/api/datastream/DataStreamBatchExecutionITCase.java
##########
@@ -180,6 +207,350 @@ public void batchSumSingleResultPerKey() throws Exception {
}
}
+ /**
+ * Verifies that all regular input is processed before keyed input.
+ *
+ * <p>Here, the first input is keyed while the second input is not keyed.
+ */
+ @Test
+ public void batchKeyedNonKeyedTwoInputOperator() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(1);
+ env.setRuntimeMode(RuntimeExecutionMode.BATCH);
+
+ DataStream<Tuple2<String, Integer>> keyedInput =
+ env.fromElements(
+ Tuple2.of("regular2", 4),
+ Tuple2.of("regular1", 3),
+ Tuple2.of("regular1", 2),
+ Tuple2.of("regular2", 1))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Tuple2<String, Integer>>forMonotonousTimestamps()
+ .withTimestampAssigner((in, ts) -> in.f1));
+
+ DataStream<Tuple2<String, Integer>> regularInput =
+ env.fromElements(
+ Tuple2.of("regular4", 4),
+ Tuple2.of("regular3", 3),
+ Tuple2.of("regular3", 2),
+ Tuple2.of("regular4", 1))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Tuple2<String, Integer>>forMonotonousTimestamps()
+ .withTimestampAssigner((in, ts) -> in.f1));
+
+ DataStream<String> result =
+ keyedInput
+ .keyBy(in -> in.f0)
+ .connect(regularInput)
+ .transform(
+ "operator",
+ BasicTypeInfo.STRING_TYPE_INFO,
+ new TwoInputIdentityOperator());
+
+ try (CloseableIterator<String> resultIterator = result.executeAndCollect()) {
+ List<String> results = CollectionUtil.iteratorToList(resultIterator);
+ assertThat(
+ results,
+ equalTo(
+ Arrays.asList(
+ "(regular4,4)",
+ "(regular3,3)",
+ "(regular3,2)",
+ "(regular4,1)",
+ "(regular1,2)",
+ "(regular1,3)",
+ "(regular2,1)",
+ "(regular2,4)")));
+ }
+ }
+
+ /**
+ * Verifies that all regular input is processed before keyed input.
+ *
+ * <p>Here, the first input is not keyed while the second input is keyed.
+ */
+ @Test
+ public void batchNonKeyedKeyedTwoInputOperator() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(1);
+ env.setRuntimeMode(RuntimeExecutionMode.BATCH);
+
+ DataStream<Tuple2<String, Integer>> keyedInput =
+ env.fromElements(
+ Tuple2.of("regular2", 4),
+ Tuple2.of("regular1", 3),
+ Tuple2.of("regular1", 2),
+ Tuple2.of("regular2", 1))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Tuple2<String, Integer>>forMonotonousTimestamps()
+ .withTimestampAssigner((in, ts) -> in.f1));
+
+ DataStream<Tuple2<String, Integer>> regularInput =
+ env.fromElements(
+ Tuple2.of("regular4", 4),
+ Tuple2.of("regular3", 3),
+ Tuple2.of("regular3", 2),
+ Tuple2.of("regular4", 1))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Tuple2<String, Integer>>forMonotonousTimestamps()
+ .withTimestampAssigner((in, ts) -> in.f1));
+
+ DataStream<String> result =
+ regularInput
+ .connect(keyedInput.keyBy(in -> in.f0))
+ .transform(
+ "operator",
+ BasicTypeInfo.STRING_TYPE_INFO,
+ new TwoInputIdentityOperator());
+
+ try (CloseableIterator<String> resultIterator = result.executeAndCollect()) {
+ List<String> results = CollectionUtil.iteratorToList(resultIterator);
+ assertThat(
+ results,
+ equalTo(
+ Arrays.asList(
+ "(regular4,4)",
+ "(regular3,3)",
+ "(regular3,2)",
+ "(regular4,1)",
+ "(regular1,2)",
+ "(regular1,3)",
+ "(regular2,1)",
+ "(regular2,4)")));
+ }
+ }
+
+ /** Verifies that all broadcast input is processed before keyed input. */
+ @Test
+ public void batchKeyedBroadcastExecution() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(1);
+ env.setRuntimeMode(RuntimeExecutionMode.BATCH);
+
+ DataStream<Tuple2<String, Integer>> bcInput =
+ env.fromElements(Tuple2.of("bc1", 1), Tuple2.of("bc2", 2), Tuple2.of("bc3", 3))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Tuple2<String, Integer>>forMonotonousTimestamps()
+ .withTimestampAssigner((in, ts) -> in.f1));
+
+ DataStream<Tuple2<String, Integer>> regularInput =
+ env.fromElements(
+ Tuple2.of("regular1", 1),
+ Tuple2.of("regular1", 2),
+ Tuple2.of("regular2", 2),
+ Tuple2.of("regular1", 3),
+ Tuple2.of("regular1", 4),
+ Tuple2.of("regular1", 3),
+ Tuple2.of("regular2", 5),
+ Tuple2.of("regular1", 5),
+ Tuple2.of("regular2", 3),
+ Tuple2.of("regular1", 3))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Tuple2<String, Integer>>forMonotonousTimestamps()
+ .withTimestampAssigner((in, ts) -> in.f1));
+
+ BroadcastStream<Tuple2<String, Integer>> broadcastStream =
+ bcInput.broadcast(STATE_DESCRIPTOR);
+
+ DataStream<String> result =
+ regularInput
+ .keyBy((input) -> input.f0)
+ .connect(broadcastStream)
+ .process(new TestKeyedBroadcastFunction());
+
+ try (CloseableIterator<String> resultIterator = result.executeAndCollect()) {
+ List<String> results = CollectionUtil.iteratorToList(resultIterator);
+ assertThat(
+ results,
+ equalTo(
+ Arrays.asList(
+ "(regular1,1): [bc2=bc2, bc1=bc1, bc3=bc3]",
+ "(regular1,2): [bc2=bc2, bc1=bc1, bc3=bc3]",
+ "(regular1,3): [bc2=bc2, bc1=bc1, bc3=bc3]",
+ "(regular1,3): [bc2=bc2, bc1=bc1, bc3=bc3]",
+ "(regular1,3): [bc2=bc2, bc1=bc1, bc3=bc3]",
+ "(regular1,4): [bc2=bc2, bc1=bc1, bc3=bc3]",
+ "(regular1,5): [bc2=bc2, bc1=bc1, bc3=bc3]",
+ "(regular2,2): [bc2=bc2, bc1=bc1, bc3=bc3]",
+ "(regular2,3): [bc2=bc2, bc1=bc1, bc3=bc3]",
+ "(regular2,5): [bc2=bc2, bc1=bc1, bc3=bc3]")));
+ }
+ }
+
+ /** Verifies that all broadcast input is processed before regular input. */
+ @Test
+ public void batchBroadcastExecution() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(1);
+ env.setRuntimeMode(RuntimeExecutionMode.BATCH);
+
+ DataStream<Tuple2<String, Integer>> bcInput =
+ env.fromElements(Tuple2.of("bc1", 1), Tuple2.of("bc2", 2), Tuple2.of("bc3", 3))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Tuple2<String, Integer>>forMonotonousTimestamps()
+ .withTimestampAssigner((in, ts) -> in.f1));
+
+ DataStream<Tuple2<String, Integer>> regularInput =
+ env.fromElements(
+ Tuple2.of("regular1", 1),
+ Tuple2.of("regular1", 2),
+ Tuple2.of("regular1", 3),
+ Tuple2.of("regular1", 4),
+ Tuple2.of("regular1", 3),
+ Tuple2.of("regular1", 5),
+ Tuple2.of("regular1", 3))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Tuple2<String, Integer>>forMonotonousTimestamps()
+ .withTimestampAssigner((in, ts) -> in.f1));
+
+ BroadcastStream<Tuple2<String, Integer>> broadcastStream =
+ bcInput.broadcast(STATE_DESCRIPTOR);
+
+ DataStream<String> result =
+ regularInput.connect(broadcastStream).process(new TestBroadcastFunction());
+
+ try (CloseableIterator<String> resultIterator = result.executeAndCollect()) {
+ List<String> results = CollectionUtil.iteratorToList(resultIterator);
+ // regular, that is non-keyed input is not sorted by timestamp. For keyed inputs
+ // this is a by-product of the grouping/sorting we use to get the keyed groups.
+ assertThat(
+ results,
+ equalTo(
+ Arrays.asList(
+ "(regular1,1): [bc2=bc2, bc1=bc1, bc3=bc3]",
+ "(regular1,2): [bc2=bc2, bc1=bc1, bc3=bc3]",
+ "(regular1,3): [bc2=bc2, bc1=bc1, bc3=bc3]",
+ "(regular1,4): [bc2=bc2, bc1=bc1, bc3=bc3]",
+ "(regular1,3): [bc2=bc2, bc1=bc1, bc3=bc3]",
+ "(regular1,5): [bc2=bc2, bc1=bc1, bc3=bc3]",
+ "(regular1,3): [bc2=bc2, bc1=bc1, bc3=bc3]")));
+ }
+ }
+
+ @Test
+ public void batchMixedKeyedAndNonKeyedTwoInputOperator() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(1);
+ env.setRuntimeMode(RuntimeExecutionMode.BATCH);
+
+ DataStream<Tuple2<String, Integer>> bcInput =
+ env.fromElements(Tuple2.of("bc3", 3), Tuple2.of("bc2", 2), Tuple2.of("bc1", 1))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Tuple2<String, Integer>>forMonotonousTimestamps()
+ .withTimestampAssigner((in, ts) -> in.f1))
+ .broadcast();
+
+ DataStream<Tuple2<String, Integer>> regularInput =
+ env.fromElements(
+ Tuple2.of("regular1", 1),
+ Tuple2.of("regular1", 2),
+ Tuple2.of("regular1", 3),
+ Tuple2.of("regular1", 4),
+ Tuple2.of("regular2", 3),
+ Tuple2.of("regular2", 5),
+ Tuple2.of("regular1", 3))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Tuple2<String, Integer>>forMonotonousTimestamps()
+ .withTimestampAssigner((in, ts) -> in.f1))
+ .keyBy(input -> input.f0);
+
+ TwoInputTransformation<Tuple2<String, Integer>, Tuple2<String, Integer>, String>
+ twoInputTransformation =
+ new TwoInputTransformation<>(
+ regularInput.getTransformation(),
+ bcInput.getTransformation(),
+ "operator",
+ new TestMixedTwoInputOperator(),
+ BasicTypeInfo.STRING_TYPE_INFO,
+ 1);
+
+ twoInputTransformation.setStateKeyType(BasicTypeInfo.STRING_TYPE_INFO);
+ twoInputTransformation.setStateKeySelectors(input -> input.f0, null);
+
+ DataStream<String> result = new DataStream<>(env, twoInputTransformation);
+
+ try (CloseableIterator<String> resultIterator = result.executeAndCollect()) {
+ List<String> results = CollectionUtil.iteratorToList(resultIterator);
+ // regular, that is non-keyed input is not sorted by timestamp. For keyed inputs
+ // this is a by-product of the grouping/sorting we use to get the keyed groups.
+ assertThat(
+ results,
+ equalTo(
+ Arrays.asList(
+ "(regular1,1): [bc3, bc2, bc1]",
+ "(regular1,2): [bc3, bc2, bc1]",
+ "(regular1,3): [bc3, bc2, bc1]",
+ "(regular1,3): [bc3, bc2, bc1]",
+ "(regular1,4): [bc3, bc2, bc1]",
+ "(regular2,3): [bc3, bc2, bc1]",
+ "(regular2,5): [bc3, bc2, bc1]")));
+ }
+ }
+
+ @Test
+ public void batchMixedKeyedAndNonKeyedMultiInputOperator() throws Exception {
+ StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
+ env.setRuntimeMode(RuntimeExecutionMode.BATCH);
+
+ DataStream<Tuple2<String, Integer>> bc1Input =
+ env.fromElements(Tuple2.of("bc3", 3), Tuple2.of("bc2", 2))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Tuple2<String, Integer>>forMonotonousTimestamps()
+ .withTimestampAssigner((in, ts) -> in.f1))
+ .broadcast();
+ DataStream<Tuple2<String, Integer>> bc2Input =
+ env.fromElements(Tuple2.of("bc1", 1))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Tuple2<String, Integer>>forMonotonousTimestamps()
+ .withTimestampAssigner((in, ts) -> in.f1))
+ .broadcast();
+ DataStream<Tuple2<String, Integer>> regularInput =
+ env.fromElements(
+ Tuple2.of("regular1", 1),
+ Tuple2.of("regular1", 2),
+ Tuple2.of("regular1", 3),
+ Tuple2.of("regular1", 4),
+ Tuple2.of("regular2", 3),
+ Tuple2.of("regular2", 5),
+ Tuple2.of("regular1", 3))
+ .assignTimestampsAndWatermarks(
+ WatermarkStrategy.<Tuple2<String, Integer>>forMonotonousTimestamps()
+ .withTimestampAssigner((in, ts) -> in.f1))
+ .keyBy(input -> input.f0);
+
+ KeyedMultipleInputTransformation<String> multipleInputTransformation =
+ new KeyedMultipleInputTransformation<>(
+ "operator",
+ mixedInputsOperatorFactory,
+ BasicTypeInfo.STRING_TYPE_INFO,
+ 1,
+ BasicTypeInfo.STRING_TYPE_INFO);
+ multipleInputTransformation.addInput(
+ regularInput.getTransformation(), input -> ((Tuple2<String, Integer>) input).f0);
+ multipleInputTransformation.addInput(bc1Input.getTransformation(), null);
+ multipleInputTransformation.addInput(bc2Input.getTransformation(), null);
+
+ DataStream<String> result =
+ new MultipleConnectedStreams(env).transform(multipleInputTransformation);
+
+ try (CloseableIterator<String> resultIterator = result.executeAndCollect()) {
+ List<String> results = CollectionUtil.iteratorToList(resultIterator);
+ // regular, that is non-keyed input is not sorted by timestamp. For keyed inputs
Review comment:
Actually the non-broadcasted input is keyed, so we should probably remove the comment.
I copied it over from one of your tests and forgot to remove the comment. The same applies for the other manually crafted two input broadcast operator.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8dd56a728ce8bf762ccd69d2d9131de62319064f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099)
* 49ca15835522bee396ea7dc88c1ab060e14bd7cb UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] kl0u commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
kl0u commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r545953329
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamMultipleInputProcessorFactory.java
##########
@@ -143,7 +150,20 @@ else if (configuredInput instanceof StreamConfig.SourceInputConfig) {
jobConfig
);
- inputs = selectableSortingInputs.getSortingInputs();
+ StreamTaskInput<?>[] sortedInputs = selectableSortingInputs.getSortedInputs();
+ StreamTaskInput<?>[] passThroughInputs = selectableSortingInputs.getPassThroughInputs();
+ int sortedIndex = 0;
+ int passThroughIndex = 0;
+ for (int i = 0; i < inputs.length; i++) {
+ if (requiresSorting(inputConfigs[i])) {
+ inputs[i] = sortedInputs[sortedIndex];
+ sortedIndex++;
+ } else {
+ inputs[i] = passThroughInputs[passThroughIndex];
+ passThroughIndex++;
+ }
+ }
+ inputs = selectableSortingInputs.getSortedInputs();
Review comment:
This line seems to be undoing what you did before, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aaa5bc44d948629278738fd441d711b32c877fba Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542)
* 73df4e8f714ac2ae08213542130d91196dc50792 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57afbcf067fb64d6e669a45accbb95aaca082345 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d2bc9331d2652a2e47038f885036b01ea936e48f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724)
* f76d4c8d68a126aaf9a52b2deb99500a76ed412d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 57afbcf067fb64d6e669a45accbb95aaca082345 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534)
* aaa5bc44d948629278738fd441d711b32c877fba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7d1707557148278604a497b07e2682b342da246 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863)
* 1e4d45934795e74502b7871d1cbad7380727ad22 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023)
* 091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6f0b8a4de3ba32b1cf407287add34dad903a68a2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770)
* ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] kl0u commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
kl0u commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r545134296
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/BroadcastConnectedStream.java
##########
@@ -205,26 +212,46 @@ public StreamExecutionEnvironment getExecutionEnvironment() {
final TypeInformation<OUT> outTypeInfo) {
Preconditions.checkNotNull(function);
- Preconditions.checkArgument(!(nonBroadcastStream instanceof KeyedStream),
+ Preconditions.checkArgument(
+ !(nonBroadcastStream instanceof KeyedStream),
"A BroadcastProcessFunction can only be used on a non-keyed stream.");
- TwoInputStreamOperator<IN1, IN2, OUT> operator =
- new CoBroadcastWithNonKeyedOperator<>(clean(function), broadcastStateDescriptors);
- return transform("Co-Process-Broadcast", outTypeInfo, operator);
+ return transform("Co-Process-Broadcast", function, outTypeInfo);
}
@Internal
private <OUT> SingleOutputStreamOperator<OUT> transform(
final String functionName,
- final TypeInformation<OUT> outTypeInfo,
- final TwoInputStreamOperator<IN1, IN2, OUT> operator) {
+ BroadcastProcessFunction<IN1, IN2, OUT> userFunction,
Review comment:
Add `final` for uniformity.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/BroadcastConnectedStream.java
##########
@@ -205,26 +212,46 @@ public StreamExecutionEnvironment getExecutionEnvironment() {
final TypeInformation<OUT> outTypeInfo) {
Preconditions.checkNotNull(function);
- Preconditions.checkArgument(!(nonBroadcastStream instanceof KeyedStream),
+ Preconditions.checkArgument(
+ !(nonBroadcastStream instanceof KeyedStream),
"A BroadcastProcessFunction can only be used on a non-keyed stream.");
- TwoInputStreamOperator<IN1, IN2, OUT> operator =
- new CoBroadcastWithNonKeyedOperator<>(clean(function), broadcastStateDescriptors);
- return transform("Co-Process-Broadcast", outTypeInfo, operator);
+ return transform("Co-Process-Broadcast", function, outTypeInfo);
}
@Internal
private <OUT> SingleOutputStreamOperator<OUT> transform(
final String functionName,
- final TypeInformation<OUT> outTypeInfo,
- final TwoInputStreamOperator<IN1, IN2, OUT> operator) {
+ BroadcastProcessFunction<IN1, IN2, OUT> userFunction,
+ final TypeInformation<OUT> outTypeInfo) {
// read the output type of the input Transforms to coax out errors about MissingTypeInfo
nonBroadcastStream.getType();
broadcastStream.getType();
final BroadcastStateTransformation<IN1, IN2, OUT> transformation =
- getBroadcastStateTransformation(functionName, outTypeInfo, operator);
+ getBroadcastStateTransformation(functionName, clean(userFunction), outTypeInfo);
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ final SingleOutputStreamOperator<OUT> returnStream =
+ new SingleOutputStreamOperator(environment, transformation);
+
+ getExecutionEnvironment().addOperator(transformation);
+ return returnStream;
+ }
+
+ @Internal
+ private <OUT> SingleOutputStreamOperator<OUT> transform(
+ final String functionName,
+ KeyedBroadcastProcessFunction<?, IN1, IN2, OUT> userFunction,
Review comment:
Add `final` for uniformity.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/BroadcastConnectedStream.java
##########
@@ -236,26 +263,32 @@ public StreamExecutionEnvironment getExecutionEnvironment() {
private <OUT> BroadcastStateTransformation<IN1, IN2, OUT> getBroadcastStateTransformation(
final String functionName,
- final TypeInformation<OUT> outTypeInfo,
- final TwoInputStreamOperator<IN1, IN2, OUT> operator) {
-
- if (nonBroadcastStream instanceof KeyedStream) {
- return BroadcastStateTransformation.forKeyedStream(
- functionName,
- (KeyedStream<IN1, ?>) nonBroadcastStream,
- broadcastStream,
- SimpleOperatorFactory.of(operator),
- outTypeInfo,
- environment.getParallelism());
- } else {
- return BroadcastStateTransformation.forNonKeyedStream(
- functionName,
- nonBroadcastStream,
- broadcastStream,
- SimpleOperatorFactory.of(operator),
- outTypeInfo,
- environment.getParallelism());
- }
+ BroadcastProcessFunction<IN1, IN2, OUT> userFunction,
+ final TypeInformation<OUT> outTypeInfo) {
+
+ return BroadcastStateTransformation.forStream(
+ functionName,
+ nonBroadcastStream,
+ broadcastStream,
+ userFunction,
+ broadcastStateDescriptors,
+ outTypeInfo,
+ environment.getParallelism());
+ }
+
+ private <KS, OUT> KeyedBroadcastStateTransformation<KS, IN1, IN2, OUT> getBroadcastStateTransformation(
+ final String functionName,
+ KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> userFunction,
Review comment:
Add `final` for uniformity.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/BroadcastStateTransformation.java
##########
@@ -53,111 +50,91 @@
@Internal
public class BroadcastStateTransformation<IN1, IN2, OUT> extends PhysicalTransformation<OUT> {
Review comment:
The two BroadcastTransformations could have a base class without the key-related fields and the function.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/BroadcastConnectedStream.java
##########
@@ -236,26 +263,32 @@ public StreamExecutionEnvironment getExecutionEnvironment() {
private <OUT> BroadcastStateTransformation<IN1, IN2, OUT> getBroadcastStateTransformation(
final String functionName,
- final TypeInformation<OUT> outTypeInfo,
- final TwoInputStreamOperator<IN1, IN2, OUT> operator) {
-
- if (nonBroadcastStream instanceof KeyedStream) {
- return BroadcastStateTransformation.forKeyedStream(
- functionName,
- (KeyedStream<IN1, ?>) nonBroadcastStream,
- broadcastStream,
- SimpleOperatorFactory.of(operator),
- outTypeInfo,
- environment.getParallelism());
- } else {
- return BroadcastStateTransformation.forNonKeyedStream(
- functionName,
- nonBroadcastStream,
- broadcastStream,
- SimpleOperatorFactory.of(operator),
- outTypeInfo,
- environment.getParallelism());
- }
+ BroadcastProcessFunction<IN1, IN2, OUT> userFunction,
+ final TypeInformation<OUT> outTypeInfo) {
+
+ return BroadcastStateTransformation.forStream(
+ functionName,
+ nonBroadcastStream,
+ broadcastStream,
+ userFunction,
+ broadcastStateDescriptors,
+ outTypeInfo,
+ environment.getParallelism());
+ }
+
Review comment:
Now the two `getBroadcastStateTransformation()` methods have no additional logic and each has a single call-site, so why not inlining them in their call-sites?
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/BroadcastStateTransformation.java
##########
@@ -53,111 +50,91 @@
@Internal
public class BroadcastStateTransformation<IN1, IN2, OUT> extends PhysicalTransformation<OUT> {
- private final Transformation<IN1> nonBroadcastStream;
+ private final BroadcastProcessFunction<IN1, IN2, OUT> userFunction;
- private final Transformation<IN2> broadcastStream;
+ private final List<MapStateDescriptor<?, ?>> broadcastStateDescriptors;
- private final StreamOperatorFactory<OUT> operatorFactory;
+ private final Transformation<IN1> inputStream;
- private final TypeInformation<?> stateKeyType;
+ private final Transformation<IN2> broadcastStream;
- private final KeySelector<IN1, ?> keySelector;
+ private ChainingStrategy chainingStrategy = ChainingStrategy.DEFAULT_CHAINING_STRATEGY;
private BroadcastStateTransformation(
final String name,
final Transformation<IN1> inputStream,
final Transformation<IN2> broadcastStream,
- final StreamOperatorFactory<OUT> operatorFactory,
- @Nullable final TypeInformation<?> keyType,
- @Nullable final KeySelector<IN1, ?> keySelector,
+ final BroadcastProcessFunction<IN1, IN2, OUT> userFunction,
+ final List<MapStateDescriptor<?, ?>> broadcastStateDescriptors,
final TypeInformation<OUT> outTypeInfo,
final int parallelism) {
super(name, outTypeInfo, parallelism);
- this.nonBroadcastStream = checkNotNull(inputStream);
+ this.inputStream = checkNotNull(inputStream);
this.broadcastStream = checkNotNull(broadcastStream);
- this.operatorFactory = checkNotNull(operatorFactory);
+ this.userFunction = userFunction;
+ this.broadcastStateDescriptors = broadcastStateDescriptors;
- this.stateKeyType = keyType;
- this.keySelector = keySelector;
- updateManagedMemoryStateBackendUseCase(keySelector != null);
+ updateManagedMemoryStateBackendUseCase(false /* not keyed */);
}
public Transformation<IN2> getBroadcastStream() {
return broadcastStream;
}
public Transformation<IN1> getNonBroadcastStream() {
- return nonBroadcastStream;
+ return inputStream;
}
- public StreamOperatorFactory<OUT> getOperatorFactory() {
- return operatorFactory;
+ public BroadcastProcessFunction<IN1, IN2, OUT> getUserFunction() {
+ return userFunction;
}
- public TypeInformation<?> getStateKeyType() {
- return stateKeyType;
+ public List<MapStateDescriptor<?, ?>> getBroadcastStateDescriptors() {
+ return broadcastStateDescriptors;
}
- public KeySelector<IN1, ?> getKeySelector() {
- return keySelector;
+ public ChainingStrategy getChainingStrategy() {
+ return chainingStrategy;
}
@Override
- public void setChainingStrategy(ChainingStrategy strategy) {
- this.operatorFactory.getChainingStrategy();
+ public void setChainingStrategy(ChainingStrategy chainingStrategy) {
+ this.chainingStrategy = checkNotNull(chainingStrategy);
}
@Override
public List<Transformation<?>> getTransitivePredecessors() {
final List<Transformation<?>> predecessors = new ArrayList<>();
predecessors.add(this);
- predecessors.add(nonBroadcastStream);
+ predecessors.add(inputStream);
predecessors.add(broadcastStream);
return predecessors;
}
@Override
public List<Transformation<?>> getInputs() {
final List<Transformation<?>> predecessors = new ArrayList<>();
- predecessors.add(nonBroadcastStream);
+ predecessors.add(inputStream);
predecessors.add(broadcastStream);
return predecessors;
}
// ------------------------------- Static Constructors -------------------------------
- public static <IN1, IN2, OUT> BroadcastStateTransformation<IN1, IN2, OUT> forNonKeyedStream(
- final String name,
- final DataStream<IN1> nonBroadcastStream,
- final BroadcastStream<IN2> broadcastStream,
- final StreamOperatorFactory<OUT> operatorFactory,
- final TypeInformation<OUT> outTypeInfo,
- final int parallelism) {
- return new BroadcastStateTransformation<>(
- name,
- checkNotNull(nonBroadcastStream).getTransformation(),
- checkNotNull(broadcastStream).getTransformation(),
- operatorFactory,
- null,
- null,
- outTypeInfo,
- parallelism);
- }
-
- public static <IN1, IN2, OUT> BroadcastStateTransformation<IN1, IN2, OUT> forKeyedStream(
+ public static <IN1, IN2, OUT> BroadcastStateTransformation<IN1, IN2, OUT> forStream(
Review comment:
We can make the constructor `public` now and remove the static constructors from here and from the `keyed` alternative as now we have two separate classes.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/KeyedBroadcastStateTransformation.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.streaming.api.datastream.BroadcastStream;
+import org.apache.flink.streaming.api.datastream.KeyedStream;
+import org.apache.flink.streaming.api.functions.co.KeyedBroadcastProcessFunction;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This is the transformation for the Broadcast State pattern. In a nutshell, this transformation
+ * allows to take a broadcasted (non-keyed) stream, connect it with another keyed or non-keyed
+ * stream, and apply a function on the resulting connected stream. This function will have access to
+ * all the elements that belong to the non-keyed, broadcasted side, as this is kept in Flink's
+ * state.
+ *
+ * <p>For more information see the
+ * <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html">
+ * Broadcast State Pattern documentation page</a>.
+ *
+ * @param <IN1> The type of the elements in the non-broadcasted input.
+ * @param <IN2> The type of the elements in the broadcasted input.
+ * @param <OUT> The type of the elements that result from this transformation.
+ */
+@Internal
+public class KeyedBroadcastStateTransformation<KS, IN1, IN2, OUT> extends PhysicalTransformation<OUT> {
+
+ private final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> userFunction;
+
+ private final List<MapStateDescriptor<?, ?>> broadcastStateDescriptors;
+
+ private final Transformation<IN1> inputStream;
+
+ private final Transformation<IN2> broadcastStream;
+
+ private final TypeInformation<?> stateKeyType;
+
+ private final KeySelector<IN1, ?> keySelector;
Review comment:
Same here.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/KeyedBroadcastStateTransformation.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.streaming.api.datastream.BroadcastStream;
+import org.apache.flink.streaming.api.datastream.KeyedStream;
+import org.apache.flink.streaming.api.functions.co.KeyedBroadcastProcessFunction;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This is the transformation for the Broadcast State pattern. In a nutshell, this transformation
+ * allows to take a broadcasted (non-keyed) stream, connect it with another keyed or non-keyed
+ * stream, and apply a function on the resulting connected stream. This function will have access to
+ * all the elements that belong to the non-keyed, broadcasted side, as this is kept in Flink's
+ * state.
+ *
+ * <p>For more information see the
+ * <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html">
+ * Broadcast State Pattern documentation page</a>.
+ *
+ * @param <IN1> The type of the elements in the non-broadcasted input.
+ * @param <IN2> The type of the elements in the broadcasted input.
+ * @param <OUT> The type of the elements that result from this transformation.
+ */
+@Internal
+public class KeyedBroadcastStateTransformation<KS, IN1, IN2, OUT> extends PhysicalTransformation<OUT> {
+
+ private final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> userFunction;
+
+ private final List<MapStateDescriptor<?, ?>> broadcastStateDescriptors;
+
+ private final Transformation<IN1> inputStream;
+
+ private final Transformation<IN2> broadcastStream;
+
+ private final TypeInformation<?> stateKeyType;
+
+ private final KeySelector<IN1, ?> keySelector;
+
+ private ChainingStrategy chainingStrategy = ChainingStrategy.DEFAULT_CHAINING_STRATEGY;
+
+ private KeyedBroadcastStateTransformation(
+ final String name,
+ final Transformation<IN1> inputStream,
+ final Transformation<IN2> broadcastStream,
+ final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> userFunction,
+ final List<MapStateDescriptor<?, ?>> broadcastStateDescriptors,
+ final TypeInformation<?> keyType,
+ final KeySelector<IN1, ?> keySelector,
+ final TypeInformation<OUT> outTypeInfo,
+ final int parallelism) {
+ super(name, outTypeInfo, parallelism);
+ this.inputStream = checkNotNull(inputStream);
+ this.broadcastStream = checkNotNull(broadcastStream);
+ this.userFunction = userFunction;
+ this.broadcastStateDescriptors = broadcastStateDescriptors;
+
+ this.stateKeyType = keyType;
+ this.keySelector = keySelector;
+ updateManagedMemoryStateBackendUseCase(true /* we have keyed state */);
+ }
+
+ public Transformation<IN2> getBroadcastStream() {
+ return broadcastStream;
+ }
+
+ public Transformation<IN1> getNonBroadcastStream() {
+ return inputStream;
+ }
+
+ public KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> getUserFunction() {
+ return userFunction;
+ }
+
+ public List<MapStateDescriptor<?, ?>> getBroadcastStateDescriptors() {
+ return broadcastStateDescriptors;
+ }
+
+ public TypeInformation<?> getStateKeyType() {
+ return stateKeyType;
+ }
+
+ public KeySelector<IN1, ?> getKeySelector() {
+ return keySelector;
+ }
+
+ public ChainingStrategy getChainingStrategy() {
+ return chainingStrategy;
+ }
+
+ @Override
+ public void setChainingStrategy(ChainingStrategy chainingStrategy) {
+ this.chainingStrategy = checkNotNull(chainingStrategy);
+ }
+
+ @Override
+ public List<Transformation<?>> getTransitivePredecessors() {
+ final List<Transformation<?>> predecessors = new ArrayList<>();
+ predecessors.add(this);
+ predecessors.add(inputStream);
+ predecessors.add(broadcastStream);
+ return predecessors;
+ }
+
+ @Override
+ public List<Transformation<?>> getInputs() {
+ final List<Transformation<?>> predecessors = new ArrayList<>();
+ predecessors.add(inputStream);
+ predecessors.add(broadcastStream);
+ return predecessors;
+ }
+
+ // ------------------------------- Static Constructors -------------------------------
+
Review comment:
Now we can remove this and make the constructor `public`, right?
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/KeyedBroadcastStateTransformation.java
##########
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.streaming.api.datastream.BroadcastStream;
+import org.apache.flink.streaming.api.datastream.KeyedStream;
+import org.apache.flink.streaming.api.functions.co.KeyedBroadcastProcessFunction;
+import org.apache.flink.streaming.api.operators.ChainingStrategy;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * This is the transformation for the Broadcast State pattern. In a nutshell, this transformation
+ * allows to take a broadcasted (non-keyed) stream, connect it with another keyed or non-keyed
+ * stream, and apply a function on the resulting connected stream. This function will have access to
+ * all the elements that belong to the non-keyed, broadcasted side, as this is kept in Flink's
+ * state.
+ *
+ * <p>For more information see the
+ * <a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html">
+ * Broadcast State Pattern documentation page</a>.
+ *
+ * @param <IN1> The type of the elements in the non-broadcasted input.
+ * @param <IN2> The type of the elements in the broadcasted input.
+ * @param <OUT> The type of the elements that result from this transformation.
+ */
+@Internal
+public class KeyedBroadcastStateTransformation<KS, IN1, IN2, OUT> extends PhysicalTransformation<OUT> {
+
+ private final KeyedBroadcastProcessFunction<KS, IN1, IN2, OUT> userFunction;
+
+ private final List<MapStateDescriptor<?, ?>> broadcastStateDescriptors;
+
+ private final Transformation<IN1> inputStream;
+
+ private final Transformation<IN2> broadcastStream;
+
+ private final TypeInformation<?> stateKeyType;
Review comment:
Now this can have `KS` as a generic type, right? (for type safety)
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java
##########
@@ -735,4 +761,10 @@ public int hashCode() {
return inputEdge.hashCode();
}
}
+
Review comment:
Why not putting it as a method in the `InputConfig` with a default implementation that returns `false`? I think that this will make the code nicer and also it will avoid checks like `inputConfig instanceof StreamConfig.NetworkInputConfig`
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamNode.java
##########
@@ -84,7 +84,7 @@
private String transformationUID;
private String userHash;
- private boolean sortedInputs = false;
+ private Map<Integer, StreamConfig.InputRequirement> inputRequirements = new HashMap<>();
Review comment:
Why not making this `final`?
Also nit-pick: I find it is nicer to initialize fields in the constructor but feel free to ignore this.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/BroadcastStateTransformationTranslator.java
##########
@@ -51,14 +69,18 @@
checkNotNull(transformation);
checkNotNull(context);
+ CoBroadcastWithNonKeyedOperator<IN1, IN2, OUT> operator = new CoBroadcastWithNonKeyedOperator<>(
Review comment:
I like that we create the operator here as now we push further down the runtime-related stuff. In the future we could have a translator that gives, for example, not an operator but sth else.
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamConfig.java
##########
@@ -238,10 +231,12 @@ private void setTypeSerializer(String key, TypeSerializer<?> typeWrapper) {
}
}
+ // TODO: this method is a bit misleading, because what it really does is setting up
Review comment:
Can't we rename it to sth more descriptive?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha closed pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha closed pull request #14312:
URL: https://github.com/apache/flink/pull/14312
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6f0b8a4de3ba32b1cf407287add34dad903a68a2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770)
* ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551441044
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/TwoInputTransformationTranslator.java
##########
@@ -46,7 +47,11 @@
transformation.getStateKeySelector1() != null
&& transformation.getStateKeySelector2() != null;
if (isKeyed) {
Review comment:
I was thinking of tests like those: https://github.com/dawidwys/flink/blob/mix-keyed-2/flink-tests/src/test/java/org/apache/flink/api/datastream/DataStreamBatchExecutionITCase.java
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
}, {
"hash" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643",
"triggerID" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"triggerType" : "PUSH"
}, {
"hash" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11662",
"triggerID" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"triggerType" : "PUSH"
}, {
"hash" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11665",
"triggerID" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"triggerType" : "PUSH"
}, {
"hash" : "f0698650b76335ee6a8a9b3079e70e853040da59",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11691",
"triggerID" : "f0698650b76335ee6a8a9b3079e70e853040da59",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f0698650b76335ee6a8a9b3079e70e853040da59 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11691)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7d1707557148278604a497b07e2682b342da246 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863)
* 1e4d45934795e74502b7871d1cbad7380727ad22 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551459508
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/TwoInputTransformationTranslator.java
##########
@@ -46,7 +47,11 @@
transformation.getStateKeySelector1() != null
&& transformation.getStateKeySelector2() != null;
if (isKeyed) {
Review comment:
Sorry, I also got started on it and should have notified you. 🙈
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f3deb69af6892fc676aa87e1d1b55981eded305 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814)
* 6d0f9c6a32d65c82e657cfcab048a98aa4445429 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849)
* 897fe1eb2bb46f63f8cea373c5b5d917070823b5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551353423
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamTwoInputProcessorFactory.java
##########
@@ -90,24 +94,51 @@
InputSelectable inputSelectable =
streamOperator instanceof InputSelectable ? (InputSelectable) streamOperator : null;
- if (streamConfig.shouldSortInputs()) {
+
+ // this is a bit verbose because we're manually handling input1 and input2
+ // TODO: extract method
+ StreamConfig.InputConfig[] inputConfigs = streamConfig.getInputs(userClassloader);
+ boolean input1IsSorted = requiresSorting(inputConfigs[0]);
+ boolean input2IsSorted = requiresSorting(inputConfigs[1]);
+
+ if (input1IsSorted || input2IsSorted) {
+ // as soon as one input requires sorting we need to treat all inputs differently, to
+ // make sure that pass-through inputs have precedence
if (inputSelectable != null) {
throw new IllegalStateException(
"The InputSelectable interface is not supported with sorting inputs");
}
- @SuppressWarnings("unchecked")
+ List<StreamTaskInput<?>> sortedTaskInputs = new ArrayList<>();
+ List<StreamTaskInput<?>> passThroughTaskInputs = new ArrayList<>();
+ int input1Index = 0;
Review comment:
fixing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073)
* 8dd56a728ce8bf762ccd69d2d9131de62319064f UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
}, {
"hash" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643",
"triggerID" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"triggerType" : "PUSH"
}, {
"hash" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11662",
"triggerID" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"triggerType" : "PUSH"
}, {
"hash" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11665",
"triggerID" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 30f17119c79e6800df9800df9cf63a700f7e2624 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11665)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8dd56a728ce8bf762ccd69d2d9131de62319064f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551391876
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/translators/TwoInputTransformationTranslator.java
##########
@@ -46,7 +47,11 @@
transformation.getStateKeySelector1() != null
&& transformation.getStateKeySelector2() != null;
if (isKeyed) {
Review comment:
sure, I will prepare one
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
}, {
"hash" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643",
"triggerID" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"triggerType" : "PUSH"
}, {
"hash" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11662",
"triggerID" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"triggerType" : "PUSH"
}, {
"hash" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11665",
"triggerID" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643)
* 84d2f162af19db4ca01d2015e81d4e71533c9a01 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11662)
* 30f17119c79e6800df9800df9cf63a700f7e2624 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11665)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551357902
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamMultipleInputProcessorFactory.java
##########
@@ -142,6 +148,7 @@ public static StreamMultipleInputProcessor create(
idx, userClassloader))
.toArray(TypeSerializer[]::new),
streamConfig.getStateKeySerializer(userClassloader),
+ new StreamTaskInput[0],
Review comment:
As for the #14324 if I remember correctly we said we will drop it and support mixed keyed/non-keyed inputs: https://github.com/apache/flink/pull/14312#discussion_r540209912
As for the multi inputs those are not exposed in a nice API at all, but I don't think there are any checks or anything that would prohibit users from mixing keyed with non-keyed inputs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551460963
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/MultiInputSortingDataInput.java
##########
@@ -107,29 +112,38 @@ private MultiInputSortingDataInput(
*/
public static class SelectableSortingInputs {
Review comment:
dropping it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 897fe1eb2bb46f63f8cea373c5b5d917070823b5 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851)
* b7d1707557148278604a497b07e2682b342da246 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 37d3b02971986addded4356b88a493b824244b2e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551346465
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/BroadcastConnectedStream.java
##########
@@ -246,28 +246,36 @@ public StreamExecutionEnvironment getExecutionEnvironment() {
return returnStream;
}
- private <OUT> BroadcastStateTransformation<IN1, IN2, OUT> getBroadcastStateTransformation(
+ @Internal
+ private <KEY, OUT> SingleOutputStreamOperator<OUT> transform(
final String functionName,
- final TypeInformation<OUT> outTypeInfo,
- final TwoInputStreamOperator<IN1, IN2, OUT> operator) {
-
- if (nonBroadcastStream instanceof KeyedStream) {
- return BroadcastStateTransformation.forKeyedStream(
- functionName,
- (KeyedStream<IN1, ?>) nonBroadcastStream,
- broadcastStream,
- SimpleOperatorFactory.of(operator),
- outTypeInfo,
- environment.getParallelism());
- } else {
- return BroadcastStateTransformation.forNonKeyedStream(
- functionName,
- nonBroadcastStream,
- broadcastStream,
- SimpleOperatorFactory.of(operator),
- outTypeInfo,
- environment.getParallelism());
- }
+ final KeyedBroadcastProcessFunction<KEY, IN1, IN2, OUT> userFunction,
+ final TypeInformation<OUT> outTypeInfo) {
+
+ // read the output type of the input Transforms to coax out errors about MissingTypeInfo
+ nonBroadcastStream.getType();
+ broadcastStream.getType();
+
+ KeyedStream<IN1, KEY> keyedInputStream = (KeyedStream<IN1, KEY>) nonBroadcastStream;
+
+ final KeyedBroadcastStateTransformation<KEY, IN1, IN2, OUT> transformation =
+ new KeyedBroadcastStateTransformation<>(
+ functionName,
Review comment:
fixing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7d1707557148278604a497b07e2682b342da246 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
dawidwys commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r551355062
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamMultipleInputProcessorFactory.java
##########
@@ -142,6 +148,7 @@ public static StreamMultipleInputProcessor create(
idx, userClassloader))
.toArray(TypeSerializer[]::new),
streamConfig.getStateKeySerializer(userClassloader),
+ new StreamTaskInput[0],
Review comment:
I don't think there is anything that prohibits mixing keyed with non-keyed inputs. Users can manually craft such Transformation. As for the #14324 didn't we say we will drop it once we merge this PR with a support for mixing such inputs?
BTW, If I remember correctly we had that discussion already ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8dd56a728ce8bf762ccd69d2d9131de62319064f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099)
* 49ca15835522bee396ea7dc88c1ab060e14bd7cb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
}, {
"hash" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643",
"triggerID" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624)
* 17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
}, {
"hash" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10851",
"triggerID" : "897fe1eb2bb46f63f8cea373c5b5d917070823b5",
"triggerType" : "PUSH"
}, {
"hash" : "b7d1707557148278604a497b07e2682b342da246",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10863",
"triggerID" : "b7d1707557148278604a497b07e2682b342da246",
"triggerType" : "PUSH"
}, {
"hash" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11023",
"triggerID" : "1e4d45934795e74502b7871d1cbad7380727ad22",
"triggerType" : "PUSH"
}, {
"hash" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11042",
"triggerID" : "091ba28c2182acfbb79b8cf0c0d6b0b2d40784ed",
"triggerType" : "PUSH"
}, {
"hash" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11073",
"triggerID" : "a006bfe64ef6b5ba5e9d553a9da2c5015ca6cf2d",
"triggerType" : "PUSH"
}, {
"hash" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11099",
"triggerID" : "8dd56a728ce8bf762ccd69d2d9131de62319064f",
"triggerType" : "PUSH"
}, {
"hash" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11599",
"triggerID" : "49ca15835522bee396ea7dc88c1ab060e14bd7cb",
"triggerType" : "PUSH"
}, {
"hash" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11624",
"triggerID" : "bd7ba18d9ef4db8a86ec315bb4ed3a333a7ac01c",
"triggerType" : "PUSH"
}, {
"hash" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11643",
"triggerID" : "17d975451b8a8d5e9f2f3c0cc97c1649d6b9708e",
"triggerType" : "PUSH"
}, {
"hash" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11662",
"triggerID" : "84d2f162af19db4ca01d2015e81d4e71533c9a01",
"triggerType" : "PUSH"
}, {
"hash" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11665",
"triggerID" : "30f17119c79e6800df9800df9cf63a700f7e2624",
"triggerType" : "PUSH"
}, {
"hash" : "f0698650b76335ee6a8a9b3079e70e853040da59",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11691",
"triggerID" : "f0698650b76335ee6a8a9b3079e70e853040da59",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 30f17119c79e6800df9800df9cf63a700f7e2624 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11665)
* f0698650b76335ee6a8a9b3079e70e853040da59 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11691)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-748024081
Thanks for the review! I addressed most comments. I had to force push because the changes would have been a bit hard to follow if I had put them on top of the existing commits.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r542271310
##########
File path: docs/dev/datastream_execution_mode.md
##########
@@ -237,6 +237,36 @@ next key.
See [FLIP-140](https://cwiki.apache.org/confluence/x/kDh4CQ) for background
information on this.
+### Order of Processing
+
+The order in which records are processed in operators or user defined functions
+(UDFs) can differ between `BATCH` and `STREAMING` execution.
+
+In `STREAMING` mode, user defined functions should not make any assumptions
+about the order of incoming records. Records are processed immediately once
+they arrive in sources.
+
+In `BATCH` execution mode, there are some operations where Flink guarantees
+order. The ordering can be a side effect of the special task scheduling,
+network shuffle, and state backend (see above) or it can be a conscious choice
+by the system.
+
+There are three general types of input that we can differentiate:
+
+- _keyed input_: input from a `KeyedStream`
+- _broadcast input_: input from a broadcast stream (see also [Broadcast
+ State]({% link dev/stream/state/broadcast_state.md %}))
+- _regular input_: input that isn't any of the above types of input
+
+These are the ordering rules for the different input types
+
+- keyed inputs are processed after all other inputs
+- broadcast inputs are processed before regular inputs
+
+As mentioned above, the keyed input will be grouped and Flink will process all
+records of a keyed group consecutively before processing the next group.
Review comment:
Perfect, thanks! I knew I was asking the right person.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6f0b8a4de3ba32b1cf407287add34dad903a68a2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] aljoscha commented on a change in pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #14312:
URL: https://github.com/apache/flink/pull/14312#discussion_r540206319
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/MultiInputSortingDataInput.java
##########
@@ -315,23 +342,40 @@ private InputStatus addNextToQueue(HeadElement reuse, DataOutput<IN> output) thr
* element of all sorting inputs. Should be used by the {@link StreamInputProcessor} to
* choose the next input to consume from.
*/
- private static class InputSelector implements InputSelectable {
+ private static class InputSelector implements InputSelectable, BoundedMultiInput {
private final CommonContext commonContext;
- private final int numberOfInputs;
+ private final int numInputs;
+ private final ConcurrentLinkedQueue<Integer> passThroughInputsIndices;
Review comment:
Could be true, I went for a safe thing but could remove it. 👌
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14312: [FLINK-20491] Support Broadcast State in BATCH execution mode
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #14312:
URL: https://github.com/apache/flink/pull/14312#issuecomment-738876739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10534",
"triggerID" : "57afbcf067fb64d6e669a45accbb95aaca082345",
"triggerType" : "PUSH"
}, {
"hash" : "aaa5bc44d948629278738fd441d711b32c877fba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10542",
"triggerID" : "aaa5bc44d948629278738fd441d711b32c877fba",
"triggerType" : "PUSH"
}, {
"hash" : "73df4e8f714ac2ae08213542130d91196dc50792",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10545",
"triggerID" : "73df4e8f714ac2ae08213542130d91196dc50792",
"triggerType" : "PUSH"
}, {
"hash" : "37d3b02971986addded4356b88a493b824244b2e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10602",
"triggerID" : "37d3b02971986addded4356b88a493b824244b2e",
"triggerType" : "PUSH"
}, {
"hash" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10724",
"triggerID" : "d2bc9331d2652a2e47038f885036b01ea936e48f",
"triggerType" : "PUSH"
}, {
"hash" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10763",
"triggerID" : "f76d4c8d68a126aaf9a52b2deb99500a76ed412d",
"triggerType" : "PUSH"
}, {
"hash" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10770",
"triggerID" : "6f0b8a4de3ba32b1cf407287add34dad903a68a2",
"triggerType" : "PUSH"
}, {
"hash" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10808",
"triggerID" : "ca3900171fb130a6e379e9bf8b4f4f4f5b4c48a4",
"triggerType" : "PUSH"
}, {
"hash" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814",
"triggerID" : "9f3deb69af6892fc676aa87e1d1b55981eded305",
"triggerType" : "PUSH"
}, {
"hash" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849",
"triggerID" : "6d0f9c6a32d65c82e657cfcab048a98aa4445429",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f3deb69af6892fc676aa87e1d1b55981eded305 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10814)
* 6d0f9c6a32d65c82e657cfcab048a98aa4445429 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10849)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org