You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/03/20 16:24:21 UTC
[GitHub] [flink] pnowojski opened a new pull request #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
pnowojski opened a new pull request #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467
This PR is based on https://github.com/apache/flink/pull/11403/ so please ignore couple of first commits.
This PR adds support for watermarks, key selector and latency markers in MultipleInputStreamOperator.
## Brief change log
Please check individual commit messages.
## Verifying this change
This change added some unit and integration tests.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (**yes** / no)
- If yes, how is the feature documented? (not applicable / docs / **JavaDocs** / not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1657728ec268ea99072c507a5cc2c959a7fe69c0 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154566399",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6577",
"triggerID" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"triggerType" : "PUSH"
}, {
"hash" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/155032711",
"triggerID" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"triggerType" : "PUSH"
}, {
"hash" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6612",
"triggerID" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
* ff6dc0129cbeb2e9d13fdef11c4c365aeae42632 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/155032711) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6612)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154566399",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6577",
"triggerID" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"triggerType" : "PUSH"
}, {
"hash" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/155032711",
"triggerID" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"triggerType" : "PUSH"
}, {
"hash" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6612",
"triggerID" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
* 3ab3a90af4620dd236e4789ff28a81bf39317dc9 Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6577)
* ff6dc0129cbeb2e9d13fdef11c4c365aeae42632 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/155032711) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6612)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154566399",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6577",
"triggerID" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
* 3ab3a90af4620dd236e4789ff28a81bf39317dc9 Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6577)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154566399",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
* e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/154566399) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154566399",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6577",
"triggerID" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"triggerType" : "PUSH"
}, {
"hash" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/155032711",
"triggerID" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"triggerType" : "PUSH"
}, {
"hash" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"status" : "CANCELED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6612",
"triggerID" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"triggerType" : "PUSH"
}, {
"hash" : "9d6fbaaa22110e59ecba8c36959d28dbfaa6d587",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9d6fbaaa22110e59ecba8c36959d28dbfaa6d587",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
* ff6dc0129cbeb2e9d13fdef11c4c365aeae42632 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/155032711) Azure: [CANCELED](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6612)
* 9d6fbaaa22110e59ecba8c36959d28dbfaa6d587 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1657728ec268ea99072c507a5cc2c959a7fe69c0 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/154287090) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski merged pull request #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
pnowojski merged pull request #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154566399",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1657728ec268ea99072c507a5cc2c959a7fe69c0 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/154287090) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456)
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
* e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/154566399) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#discussion_r397445760
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/KeyedMultipleInputTransformation.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * {@link AbstractMultipleInputTransformation} implementation for keyed streams.
+ */
+@Internal
+public class KeyedMultipleInputTransformation<OUT> extends AbstractMultipleInputTransformation<OUT> {
Review comment:
Okay, that makes sense. I haven't spotted that difference and I don't see a good workaround for that (except ugly generics). Related remark, how about makes the `addInput`s builder style with self return?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#discussion_r396988936
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/AbstractStreamOperatorV2Test.java
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.operators;
+
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.BasicTypeInfo;
+import org.apache.flink.api.common.typeutils.base.StringSerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness;
+
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * Tests for the facilities provided by {@link AbstractStreamOperatorV2}.
+ */
+public class AbstractStreamOperatorV2Test extends AbstractStreamOperatorTest {
+ @Override
+ protected KeyedOneInputStreamOperatorTestHarness<Integer, Tuple2<Integer, String>, String> createTestHarness(
+ int maxParalelism,
+ int numSubtasks,
+ int subtaskIndex) throws Exception {
+ return new KeyedOneInputStreamOperatorTestHarness<>(
+ new TestOperatorFactory(),
+ new TestKeySelector(),
+ BasicTypeInfo.INT_TYPE_INFO,
+ maxParalelism,
+ numSubtasks,
+ subtaskIndex);
+ }
+
+ private static class TestOperatorFactory extends AbstractStreamOperatorFactory<String> {
+ @Override
+ public <T extends StreamOperator<String>> T createStreamOperator(StreamOperatorParameters<String> parameters) {
+ return (T) new TestOperator(parameters);
+ }
+
+ @Override
+ public Class<? extends StreamOperator> getStreamOperatorClass(ClassLoader classLoader) {
+ return TestOperator.class;
+ }
+ }
+
+ /**
+ * Testing operator that can respond to commands by either setting/deleting state, emitting
+ * state or setting timers.
+ */
+ private static class TestOperator
Review comment:
`SingleInputTestOperator`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
pnowojski commented on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-604262485
Azure test failure is unrelated, travis green. Merging. Thanks for the review @AHeise .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601791475
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 1657728ec268ea99072c507a5cc2c959a7fe69c0 (Fri Mar 20 16:27:23 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154566399",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6577",
"triggerID" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"triggerType" : "PUSH"
}, {
"hash" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/155032711",
"triggerID" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"triggerType" : "PUSH"
}, {
"hash" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"status" : "CANCELED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6612",
"triggerID" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
* ff6dc0129cbeb2e9d13fdef11c4c365aeae42632 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/155032711) Azure: [CANCELED](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6612)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#discussion_r396984936
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/MultipleInputStreamTaskTest.java
##########
@@ -355,8 +377,235 @@ public void testInputFairness() throws Exception {
}
}
- // This must only be used in one test, otherwise the static fields will be changed
- // by several tests concurrently
+ /**
+ * This test verifies that watermarks and stream statuses are correctly forwarded. This also checks whether
+ * watermarks are forwarded only when we have received watermarks from all inputs. The
+ * forwarded watermark must be the minimum of the watermarks of all active inputs.
+ */
+ @Test
+ public void testWatermarkAndStreamStatusForwarding() throws Exception {
+ try (StreamTaskMailboxTestHarness<String> testHarness =
+ new MultipleInputStreamTaskTestHarnessBuilder<>(MultipleInputStreamTask::new, BasicTypeInfo.STRING_TYPE_INFO)
+ .addInput(BasicTypeInfo.STRING_TYPE_INFO, 2)
+ .addInput(BasicTypeInfo.INT_TYPE_INFO, 2)
+ .addInput(BasicTypeInfo.DOUBLE_TYPE_INFO, 2)
+ .setupOutputForSingletonOperatorChain(new MapToStringMultipleInputOperatorFactory())
+ .build()) {
+ ArrayDeque<Object> expectedOutput = new ArrayDeque<>();
+
+ long initialTime = 0L;
+
+ testHarness.processElement(new Watermark(initialTime), 0, 0);
+ testHarness.processElement(new Watermark(initialTime), 0, 1);
+ testHarness.processElement(new Watermark(initialTime), 1, 0);
+ testHarness.processElement(new Watermark(initialTime), 1, 1);
+
+ testHarness.processElement(new Watermark(initialTime), 2, 0);
+
+ assertThat(testHarness.getOutput(), IsEmptyCollection.empty());
+
+ testHarness.processElement(new Watermark(initialTime), 2, 1);
+
+ // now the watermark should have propagated, Map simply forward Watermarks
+ expectedOutput.add(new Watermark(initialTime));
+ assertThat(testHarness.getOutput(), contains(expectedOutput.toArray()));
+
+ // contrary to checkpoint barriers these elements are not blocked by watermarks
+ testHarness.processElement(new StreamRecord<>("Hello", initialTime), 0, 0);
+ testHarness.processElement(new StreamRecord<>(42, initialTime), 1, 1);
+ expectedOutput.add(new StreamRecord<>("Hello", initialTime));
+ expectedOutput.add(new StreamRecord<>("42", initialTime));
+
+ assertThat(testHarness.getOutput(), contains(expectedOutput.toArray()));
+
+ testHarness.processElement(new Watermark(initialTime + 4), 0, 0);
+ testHarness.processElement(new Watermark(initialTime + 3), 0, 1);
+ testHarness.processElement(new Watermark(initialTime + 3), 1, 0);
+ testHarness.processElement(new Watermark(initialTime + 4), 1, 1);
+ testHarness.processElement(new Watermark(initialTime + 3), 2, 0);
+ testHarness.processElement(new Watermark(initialTime + 2), 2, 1);
+
+ // check whether we get the minimum of all the watermarks, this must also only occur in
+ // the output after the two StreamRecords
+ expectedOutput.add(new Watermark(initialTime + 2));
+ assertThat(testHarness.getOutput(), contains(expectedOutput.toArray()));
+
+ // advance watermark from one of the inputs, now we should get a new one since the
+ // minimum increases
+ testHarness.processElement(new Watermark(initialTime + 4), 2, 1);
+ expectedOutput.add(new Watermark(initialTime + 3));
+ assertThat(testHarness.getOutput(), contains(expectedOutput.toArray()));
+
+ // advance the other two inputs, now we should get a new one since the
+ // minimum increases again
+ testHarness.processElement(new Watermark(initialTime + 4), 0, 1);
+ testHarness.processElement(new Watermark(initialTime + 4), 1, 0);
+ testHarness.processElement(new Watermark(initialTime + 4), 2, 0);
+ expectedOutput.add(new Watermark(initialTime + 4));
+ assertThat(testHarness.getOutput(), contains(expectedOutput.toArray()));
+
+ // test whether idle input channels are acknowledged correctly when forwarding watermarks
+ testHarness.processElement(StreamStatus.IDLE, 0, 1);
+ testHarness.processElement(StreamStatus.IDLE, 1, 1);
+ testHarness.processElement(StreamStatus.IDLE, 2, 0);
+ testHarness.processElement(new Watermark(initialTime + 6), 0, 0);
+ testHarness.processElement(new Watermark(initialTime + 6), 1, 0);
+ testHarness.processElement(new Watermark(initialTime + 5), 2, 1); // this watermark should be advanced first
+ testHarness.processElement(StreamStatus.IDLE, 2, 1); // once this is acknowledged,
+
+ // watermark (initial + 6) should be forwarded
Review comment:
comment is contradicting assertion
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#discussion_r397129053
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/KeyedMultipleInputTransformation.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * {@link AbstractMultipleInputTransformation} implementation for keyed streams.
+ */
+@Internal
+public class KeyedMultipleInputTransformation<OUT> extends AbstractMultipleInputTransformation<OUT> {
Review comment:
It could (and initially it was like that), but in that case wouldn't `MultipleConnectedStreams` look strange?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1657728ec268ea99072c507a5cc2c959a7fe69c0 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/154287090) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456)
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154566399",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6577",
"triggerID" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"triggerType" : "PUSH"
}, {
"hash" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
* 3ab3a90af4620dd236e4789ff28a81bf39317dc9 Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6577)
* ff6dc0129cbeb2e9d13fdef11c4c365aeae42632 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#discussion_r396990158
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/util/OneInputStreamOperatorTestHarness.java
##########
@@ -40,6 +48,9 @@
public class OneInputStreamOperatorTestHarness<IN, OUT>
Review comment:
These refactorings could probably also go into previous commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#discussion_r397441916
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/KeyedMultipleInputTransformation.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * {@link AbstractMultipleInputTransformation} implementation for keyed streams.
+ */
+@Internal
+public class KeyedMultipleInputTransformation<OUT> extends AbstractMultipleInputTransformation<OUT> {
Review comment:
Ok, I remember the real reason why I introduced `Abstract` - keyed and non-keyed have different `addInput` methods, with different signature. If we make `KeyedMultipleInputTransformation extends MultipleInputTransformation`, we would drop compile time checks for correct `addInput` method calls in favour of runtime checks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#discussion_r397441916
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/KeyedMultipleInputTransformation.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * {@link AbstractMultipleInputTransformation} implementation for keyed streams.
+ */
+@Internal
+public class KeyedMultipleInputTransformation<OUT> extends AbstractMultipleInputTransformation<OUT> {
Review comment:
Ok, I remember the real reason why I introduced `Abstract` - keyed and non-keyed have different `addInput` methods, with different signature. If we make `KeyedMultipleInputTransformation extends MultipleInputTransformation`, we would drop compile time checks for correct `addInput` method calls, which is the main reason why there are two classes in the first place.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154566399",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
* e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/154566399) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#discussion_r397127890
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/util/OneInputStreamOperatorTestHarness.java
##########
@@ -40,6 +48,9 @@
public class OneInputStreamOperatorTestHarness<IN, OUT>
Review comment:
Previous commit has a small value on it's own and it's pure refactor. Changes in this file are adding new feature, which are required required by `AbstractStreamOperatorV2Test`, while without it, they are a dead/unnecessary code. (That's why I bundled them in this commit not the previous one)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154566399",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6577",
"triggerID" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
* e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/154566399) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519)
* 3ab3a90af4620dd236e4789ff28a81bf39317dc9 Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6577)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1657728ec268ea99072c507a5cc2c959a7fe69c0 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/154287090) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
pnowojski commented on a change in pull request #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#discussion_r397129466
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/MultipleInputStreamTaskTest.java
##########
@@ -385,12 +386,12 @@ public void testInputFairness() throws Exception {
@Test
public void testWatermarkAndStreamStatusForwarding() throws Exception {
try (StreamTaskMailboxTestHarness<String> testHarness =
- new MultipleInputStreamTaskTestHarnessBuilder<>(MultipleInputStreamTask::new, BasicTypeInfo.STRING_TYPE_INFO)
Review comment:
Ops, you are right. It should have been squashed with something else. It can not be in the master, as the check style would fail the build.
Fixed, previous commit in this PR was introducing this change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154566399",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6577",
"triggerID" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"triggerType" : "PUSH"
}, {
"hash" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/155032711",
"triggerID" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"triggerType" : "PUSH"
}, {
"hash" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6612",
"triggerID" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"triggerType" : "PUSH"
}, {
"hash" : "9d6fbaaa22110e59ecba8c36959d28dbfaa6d587",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/155098857",
"triggerID" : "9d6fbaaa22110e59ecba8c36959d28dbfaa6d587",
"triggerType" : "PUSH"
}, {
"hash" : "9d6fbaaa22110e59ecba8c36959d28dbfaa6d587",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6632",
"triggerID" : "9d6fbaaa22110e59ecba8c36959d28dbfaa6d587",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
* 9d6fbaaa22110e59ecba8c36959d28dbfaa6d587 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/155098857) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6632)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154566399",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6577",
"triggerID" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"triggerType" : "PUSH"
}, {
"hash" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/155032711",
"triggerID" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"triggerType" : "PUSH"
}, {
"hash" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6612",
"triggerID" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"triggerType" : "PUSH"
}, {
"hash" : "9d6fbaaa22110e59ecba8c36959d28dbfaa6d587",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/155098857",
"triggerID" : "9d6fbaaa22110e59ecba8c36959d28dbfaa6d587",
"triggerType" : "PUSH"
}, {
"hash" : "9d6fbaaa22110e59ecba8c36959d28dbfaa6d587",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6632",
"triggerID" : "9d6fbaaa22110e59ecba8c36959d28dbfaa6d587",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
* 9d6fbaaa22110e59ecba8c36959d28dbfaa6d587 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/155098857) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6632)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#discussion_r397411077
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/KeyedMultipleInputTransformation.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * {@link AbstractMultipleInputTransformation} implementation for keyed streams.
+ */
+@Internal
+public class KeyedMultipleInputTransformation<OUT> extends AbstractMultipleInputTransformation<OUT> {
Review comment:
Hm I cannot follow. `MultipleConnectedStreams` is taking a `MultipleInputTransformation` being keyed or not, so not sure what is strange looking.
The main question to answer is if the hierarchy is proper: is `KeyedMultipleInputTransformation` really a `MultipleInputTransformation`? I'd say it is.
I find the instanceof checks against `AbstractMultipleInputTransformation` stranger.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154566399",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6577",
"triggerID" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"triggerType" : "PUSH"
}, {
"hash" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/155032711",
"triggerID" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"triggerType" : "PUSH"
}, {
"hash" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"status" : "CANCELED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6612",
"triggerID" : "ff6dc0129cbeb2e9d13fdef11c4c365aeae42632",
"triggerType" : "PUSH"
}, {
"hash" : "9d6fbaaa22110e59ecba8c36959d28dbfaa6d587",
"status" : "PENDING",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/155098857",
"triggerID" : "9d6fbaaa22110e59ecba8c36959d28dbfaa6d587",
"triggerType" : "PUSH"
}, {
"hash" : "9d6fbaaa22110e59ecba8c36959d28dbfaa6d587",
"status" : "PENDING",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6632",
"triggerID" : "9d6fbaaa22110e59ecba8c36959d28dbfaa6d587",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
* ff6dc0129cbeb2e9d13fdef11c4c365aeae42632 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/155032711) Azure: [CANCELED](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6612)
* 9d6fbaaa22110e59ecba8c36959d28dbfaa6d587 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/155098857) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6632)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1657728ec268ea99072c507a5cc2c959a7fe69c0 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/154287090) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#discussion_r397004890
##########
File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/KeyedMultipleInputTransformation.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.transformations;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.dag.Transformation;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.streaming.api.operators.StreamOperatorFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * {@link AbstractMultipleInputTransformation} implementation for keyed streams.
+ */
+@Internal
+public class KeyedMultipleInputTransformation<OUT> extends AbstractMultipleInputTransformation<OUT> {
Review comment:
Can't that extend `MultipleInputTransformation` to get rid of `AbstractMultipleInputTransformation`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#discussion_r397000607
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/MultipleInputStreamTaskTest.java
##########
@@ -385,12 +386,12 @@ public void testInputFairness() throws Exception {
@Test
public void testWatermarkAndStreamStatusForwarding() throws Exception {
try (StreamTaskMailboxTestHarness<String> testHarness =
- new MultipleInputStreamTaskTestHarnessBuilder<>(MultipleInputStreamTask::new, BasicTypeInfo.STRING_TYPE_INFO)
Review comment:
Can you double-check if this code is already in master or if the formatting fixes could be squashed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1657728ec268ea99072c507a5cc2c959a7fe69c0 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/154287090) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456)
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
* e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
AHeise commented on a change in pull request #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#discussion_r396985318
##########
File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/MultipleInputStreamTaskTest.java
##########
@@ -355,8 +377,235 @@ public void testInputFairness() throws Exception {
}
}
- // This must only be used in one test, otherwise the static fields will be changed
- // by several tests concurrently
+ /**
+ * This test verifies that watermarks and stream statuses are correctly forwarded. This also checks whether
+ * watermarks are forwarded only when we have received watermarks from all inputs. The
+ * forwarded watermark must be the minimum of the watermarks of all active inputs.
+ */
+ @Test
+ public void testWatermarkAndStreamStatusForwarding() throws Exception {
+ try (StreamTaskMailboxTestHarness<String> testHarness =
+ new MultipleInputStreamTaskTestHarnessBuilder<>(MultipleInputStreamTask::new, BasicTypeInfo.STRING_TYPE_INFO)
+ .addInput(BasicTypeInfo.STRING_TYPE_INFO, 2)
+ .addInput(BasicTypeInfo.INT_TYPE_INFO, 2)
+ .addInput(BasicTypeInfo.DOUBLE_TYPE_INFO, 2)
+ .setupOutputForSingletonOperatorChain(new MapToStringMultipleInputOperatorFactory())
+ .build()) {
+ ArrayDeque<Object> expectedOutput = new ArrayDeque<>();
+
+ long initialTime = 0L;
+
+ testHarness.processElement(new Watermark(initialTime), 0, 0);
+ testHarness.processElement(new Watermark(initialTime), 0, 1);
+ testHarness.processElement(new Watermark(initialTime), 1, 0);
+ testHarness.processElement(new Watermark(initialTime), 1, 1);
+
+ testHarness.processElement(new Watermark(initialTime), 2, 0);
+
+ assertThat(testHarness.getOutput(), IsEmptyCollection.empty());
+
+ testHarness.processElement(new Watermark(initialTime), 2, 1);
+
+ // now the watermark should have propagated, Map simply forward Watermarks
+ expectedOutput.add(new Watermark(initialTime));
+ assertThat(testHarness.getOutput(), contains(expectedOutput.toArray()));
+
+ // contrary to checkpoint barriers these elements are not blocked by watermarks
+ testHarness.processElement(new StreamRecord<>("Hello", initialTime), 0, 0);
+ testHarness.processElement(new StreamRecord<>(42, initialTime), 1, 1);
+ expectedOutput.add(new StreamRecord<>("Hello", initialTime));
+ expectedOutput.add(new StreamRecord<>("42", initialTime));
+
+ assertThat(testHarness.getOutput(), contains(expectedOutput.toArray()));
+
+ testHarness.processElement(new Watermark(initialTime + 4), 0, 0);
+ testHarness.processElement(new Watermark(initialTime + 3), 0, 1);
+ testHarness.processElement(new Watermark(initialTime + 3), 1, 0);
+ testHarness.processElement(new Watermark(initialTime + 4), 1, 1);
+ testHarness.processElement(new Watermark(initialTime + 3), 2, 0);
+ testHarness.processElement(new Watermark(initialTime + 2), 2, 1);
+
+ // check whether we get the minimum of all the watermarks, this must also only occur in
+ // the output after the two StreamRecords
+ expectedOutput.add(new Watermark(initialTime + 2));
+ assertThat(testHarness.getOutput(), contains(expectedOutput.toArray()));
+
+ // advance watermark from one of the inputs, now we should get a new one since the
+ // minimum increases
+ testHarness.processElement(new Watermark(initialTime + 4), 2, 1);
+ expectedOutput.add(new Watermark(initialTime + 3));
+ assertThat(testHarness.getOutput(), contains(expectedOutput.toArray()));
+
+ // advance the other two inputs, now we should get a new one since the
+ // minimum increases again
+ testHarness.processElement(new Watermark(initialTime + 4), 0, 1);
+ testHarness.processElement(new Watermark(initialTime + 4), 1, 0);
+ testHarness.processElement(new Watermark(initialTime + 4), 2, 0);
+ expectedOutput.add(new Watermark(initialTime + 4));
+ assertThat(testHarness.getOutput(), contains(expectedOutput.toArray()));
+
+ // test whether idle input channels are acknowledged correctly when forwarding watermarks
+ testHarness.processElement(StreamStatus.IDLE, 0, 1);
Review comment:
Idle tests should be a separate test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11467:
[FLINK-16317][operators] Provide support for watermarks,
key selector and latency marker in MultipleInputStreamOperator
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #11467: [FLINK-16317][operators] Provide support for watermarks, key selector and latency marker in MultipleInputStreamOperator
URL: https://github.com/apache/flink/pull/11467#issuecomment-601798473
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6456",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"status" : "DELETED",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154287090",
"triggerID" : "1657728ec268ea99072c507a5cc2c959a7fe69c0",
"triggerType" : "PUSH"
}, {
"hash" : "81e17c31a837e19c544c85672354f9681b8a8205",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81e17c31a837e19c544c85672354f9681b8a8205",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "SUCCESS",
"url" : "https://travis-ci.com/github/flink-ci/flink/builds/154566399",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519",
"triggerID" : "e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57",
"triggerType" : "PUSH"
}, {
"hash" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3ab3a90af4620dd236e4789ff28a81bf39317dc9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81e17c31a837e19c544c85672354f9681b8a8205 UNKNOWN
* e4659f0c4ef7afa42a6a67e3c2e1eeb73db02e57 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/154566399) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6519)
* 3ab3a90af4620dd236e4789ff28a81bf39317dc9 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services