You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/18 11:21:15 UTC
[GitHub] [beam] rehmanmuradali opened a new pull request #11154: [BEAM-1819]
Key should be available in @OnTimer methods
rehmanmuradali opened a new pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601225409
> Could we ensure that the FnApiDoFnRunner implementation for the key is completed as well?
> I think the type check for `@KeyId` breaks if the key is generic. I was trying this out and got:
>
> ```
> java.lang.IllegalArgumentException: CombinerDoFn, @OnTimer onTriggerTimer(BoundedWindow, WindowedKey, CombiningState, CombiningState, ValueState, CombiningState, BagState, OutputReceiver): @Key argument is expected to be type of WindowedKey<K>, but found class WindowedKey
> ```
Thank you @steveniemitz for catching this bug. Can you please mention how can I reproduce it so that I can fix this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612852796
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601197747
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-611777030
run flink validatesrunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-611777072
run spark validatesrunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r396604278
##########
File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
##########
@@ -1155,6 +1156,10 @@ public Instant getStateStartTime() {
// Note that TimerOrElementCoder is a backwards-compatibility class
// that is really a FakeKeyedWorkItemCoder
Coder<?> valueCoder = ((WindowedValueCoder<?>) readCoder).getValueCoder();
+
+ if (valueCoder instanceof KvCoder<?, ?>) {
+ return ((KvCoder<?, ?>) valueCoder).getKeyCoder();
+ }
Review comment:
without it, the key passed into the `WindmillStateInternals` constructor is always null. This coder is used to deserialize the user key from the work item key here:
https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java#L1340
It doesn't look like it was ever really used before other than for debug messages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601220660
Run Gearpump ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607102582
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607103654
Run Java Flink PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r403549672
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/BufferingDoFnRunner.java
##########
@@ -125,8 +129,8 @@ public void processElement(WindowedValue<InputT> elem) {
Instant outputTimestamp,
TimeDomain timeDomain) {
currentBufferingElementsHandler.buffer(
- new BufferedElements.Timer(
- timerId, timerFamilyId, window, timestamp, outputTimestamp, timeDomain));
+ new BufferedElements.Timer<>(
+ timerId, timerFamilyId, key, window, timestamp, outputTimestamp, timeDomain));
Review comment:
On a closer look, I think adding to the Coder is the wrong way to preserve the key. DoFnOperator has a stateInternals() which should provide access to the current key, and that way we don't have to encode the key again.
Sorry for the bad suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r410250638
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
##########
@@ -837,11 +837,15 @@ public void onProcessingTime(InternalTimer<ByteBuffer, TimerData> timer) {
// allow overriding this in ExecutableStageDoFnOperator to set the key context
protected void fireTimerInternal(Object key, TimerData timerData) {
- fireTimer(timerData);
+ if (key instanceof java.nio.ByteBuffer) {
+ key = FlinkKeyUtils.decodeKey((ByteBuffer) key, keyCoder);
Review comment:
@mxm should I use a different interface to get the encoded key? Isn't timer.getKey was supposed to return ByteBuffer always?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r397528927
##########
File path: runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
##########
@@ -686,6 +697,11 @@ public TimeDomain timeDomain() {
return timeDomain;
}
+ @Override
+ public Object key() {
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607104209
Run Java Spark PortableValidatesRunner Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali removed a comment on issue #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-611977429
> There appears to be a compilation failure in Java
@reuvenlax fixed.
> There appears to be a compilation failure in Java
@reuvenlax fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r410197770
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
##########
@@ -837,11 +837,15 @@ public void onProcessingTime(InternalTimer<ByteBuffer, TimerData> timer) {
// allow overriding this in ExecutableStageDoFnOperator to set the key context
protected void fireTimerInternal(Object key, TimerData timerData) {
- fireTimer(timerData);
+ if (key instanceof java.nio.ByteBuffer) {
+ key = FlinkKeyUtils.decodeKey((ByteBuffer) key, keyCoder);
+ }
+
+ fireTimer(key, timerData);
}
// allow overriding this in WindowDoFnOperator
- protected void fireTimer(TimerData timerData) {
+ protected void fireTimer(Object key, TimerData timerData) {
Review comment:
```suggestion
protected void fireTimer(ByteBuffer key, TimerData timerData) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-609095883
can you elaborate? What would be backwards incompatible? Is removing state cells unsupported for updates?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614087227
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612853194
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r397529012
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
##########
@@ -218,6 +218,7 @@ private void fireTimer(TimerInternals.TimerData timer, DoFnRunner<KV<K, V>, Outp
doFnRunner.onTimer(
timer.getTimerId(),
timer.getTimerFamilyId(),
+ "",
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614087306
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612853780
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607103138
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r409681355
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
##########
@@ -837,11 +837,15 @@ public void onProcessingTime(InternalTimer<ByteBuffer, TimerData> timer) {
// allow overriding this in ExecutableStageDoFnOperator to set the key context
protected void fireTimerInternal(Object key, TimerData timerData) {
- fireTimer(timerData);
+ if (key instanceof java.nio.ByteBuffer) {
+ key = FlinkKeyUtils.decodeKey((ByteBuffer) key, keyCoder);
Review comment:
@mxm Can you take a quick look and verify that this key handling is correct here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607103371
Run Samza ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612835446
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev removed a comment on issue #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601197519
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-613523936
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601280654
so I think this was enough to get it working in dataflow:
```
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java b/runners/google-cloud-dataflow-j
ava/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java
index 2c96736..96ce156 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java
@@ -373,7 +373,7 @@ public class SimpleParDoFn<InputT, OutputT> implements ParDoFn {
fnRunner.onTimer(
timer.getTimerId(),
timer.getTimerFamilyId(),
- "",
+ this.stepContext.stateInternals().getKey(),
window,
timer.getTimestamp(),
timer.getOutputTimestamp(),
diff --git a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java b/runners/google-cloud-
dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
index 1ecdd50..04e4b15 100644
--- a/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
+++ b/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
@@ -123,6 +123,7 @@ import org.apache.beam.runners.dataflow.worker.windmill.WindmillServerStub.Commi
import org.apache.beam.runners.dataflow.worker.windmill.WindmillServerStub.GetWorkStream;
import org.apache.beam.runners.dataflow.worker.windmill.WindmillServerStub.StreamPool;
import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.extensions.gcp.util.Transport;
import org.apache.beam.sdk.fn.IdGenerator;
import org.apache.beam.sdk.fn.IdGenerators;
@@ -1365,6 +1366,10 @@ public class StreamingDataflowWorker {
// Note that TimerOrElementCoder is a backwards-compatibility class
// that is really a FakeKeyedWorkItemCoder
Coder<?> valueCoder = ((WindowedValueCoder<?>) readCoder).getValueCoder();
+
+ if (valueCoder instanceof KvCoder<?, ?>) {
+ return ((KvCoder<?, ?>) valueCoder).getKeyCoder();
+ }
if (!(valueCoder instanceof WindmillKeyedWorkItem.FakeKeyedWorkItemCoder<?, ?>)) {
return null;
}
```
cc @reuvenlax on the worker changes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614087696
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r396604278
##########
File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
##########
@@ -1155,6 +1156,10 @@ public Instant getStateStartTime() {
// Note that TimerOrElementCoder is a backwards-compatibility class
// that is really a FakeKeyedWorkItemCoder
Coder<?> valueCoder = ((WindowedValueCoder<?>) readCoder).getValueCoder();
+
+ if (valueCoder instanceof KvCoder<?, ?>) {
+ return ((KvCoder<?, ?>) valueCoder).getKeyCoder();
+ }
Review comment:
without it, the key passed into the `WindmillStateInternals` constructor is always null. It doesn't look like it was ever really used before other than for debug messages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r410239653
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
##########
@@ -837,11 +837,15 @@ public void onProcessingTime(InternalTimer<ByteBuffer, TimerData> timer) {
// allow overriding this in ExecutableStageDoFnOperator to set the key context
protected void fireTimerInternal(Object key, TimerData timerData) {
- fireTimer(timerData);
+ if (key instanceof java.nio.ByteBuffer) {
+ key = FlinkKeyUtils.decodeKey((ByteBuffer) key, keyCoder);
Review comment:
Isn't `timer.getKey()` a new interface? Keys are always internally stored encoded as ByteBuffer. The keys itself can of course be of any type. The test might not instrument the code correctly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601197519
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614087306
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-611777422
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612852886
mmmm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607103944
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612852886
mmmm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607103585
Run Dataflow PortabilityApi ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mwalenia commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
mwalenia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601684571
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601219132
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614115439
R: @reuvenlax
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614738955
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612180722
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601219318
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601218708
Run Direct ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601198848
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r395897308
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
##########
@@ -847,6 +847,7 @@ protected void fireTimer(InternalTimer<ByteBuffer, TimerData> timer) {
pushbackDoFnRunner.onTimer(
timerData.getTimerId(),
timerData.getTimerFamilyId(),
+ "",
Review comment:
Also should be easy to do here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614091721
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614087227
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev removed a comment on issue #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601198377
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601220446
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612115918
run flink validatesrunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601199613
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612180607
run dataflow validatesrunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601208448
Run Dataflow ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612853852
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601197814
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607103492
Run Spark StructuredStreaming ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r410224185
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
##########
@@ -837,11 +837,15 @@ public void onProcessingTime(InternalTimer<ByteBuffer, TimerData> timer) {
// allow overriding this in ExecutableStageDoFnOperator to set the key context
protected void fireTimerInternal(Object key, TimerData timerData) {
- fireTimer(timerData);
+ if (key instanceof java.nio.ByteBuffer) {
+ key = FlinkKeyUtils.decodeKey((ByteBuffer) key, keyCoder);
Review comment:
@mxm, upon debugging, I noticed that timer.getKey() returns String, Integer, as well as ByteBuffer. That's why when I changed the method signature, DoFnOperatorTest failed with ClassCastException.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601200225
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607103371
Run Samza ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mwalenia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
mwalenia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601684571
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r402611665
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/BufferedElements.java
##########
@@ -90,7 +90,8 @@ public int hashCode() {
@Override
public void processWith(DoFnRunner doFnRunner) {
- doFnRunner.onTimer(timerId, timerFamilyId, window, timestamp, outputTimestamp, timeDomain);
Review comment:
Can we pass the key through here? We might need to include it in the coder for BufferedElements.Timer
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r404234652
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/BufferingDoFnRunner.java
##########
@@ -125,8 +129,8 @@ public void processElement(WindowedValue<InputT> elem) {
Instant outputTimestamp,
TimeDomain timeDomain) {
currentBufferingElementsHandler.buffer(
- new BufferedElements.Timer(
- timerId, timerFamilyId, window, timestamp, outputTimestamp, timeDomain));
+ new BufferedElements.Timer<>(
+ timerId, timerFamilyId, key, window, timestamp, outputTimestamp, timeDomain));
Review comment:
@reuvenlax done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614091721
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601221088
I think the type check for `@KeyId` breaks if the key is generic. I was trying this out and got:
```
java.lang.IllegalArgumentException: CombinerDoFn, @OnTimer onTriggerTimer(BoundedWindow, WindowedKey, CombiningState, CombiningState, ValueState, CombiningState, BagState, OutputReceiver): @Key argument is expected to be type of WindowedKey<K>, but found class WindowedKey
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601199144
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev removed a comment on issue #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601199613
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev removed a comment on issue #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601199717
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612853600
Run Dataflow Validatesrunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r395633155
##########
File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java
##########
@@ -361,6 +361,7 @@ private void processUserTimer(TimerData timer) throws Exception {
fnRunner.onTimer(
timer.getTimerId(),
timer.getTimerFamilyId(),
+ this.stepContext.stateInternals().getKey(),
Review comment:
you'll need the change to StreamingDataflowWorker as well, without it, the key will always be null.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-609095558
@steveniemitz It's possible that might be a backwards incompatible change (for runners that support update)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-600671323
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614739052
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607102432
Run Dataflow ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev removed a comment on issue #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601200096
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601219003
Run Direct ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r410199788
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
##########
@@ -837,11 +837,15 @@ public void onProcessingTime(InternalTimer<ByteBuffer, TimerData> timer) {
// allow overriding this in ExecutableStageDoFnOperator to set the key context
protected void fireTimerInternal(Object key, TimerData timerData) {
- fireTimer(timerData);
+ if (key instanceof java.nio.ByteBuffer) {
+ key = FlinkKeyUtils.decodeKey((ByteBuffer) key, keyCoder);
+ }
+
+ fireTimer(key, timerData);
}
// allow overriding this in WindowDoFnOperator
- protected void fireTimer(TimerData timerData) {
+ protected void fireTimer(Object key, TimerData timerData) {
Review comment:
If you decide the decode the key already in the calling method, then use the key type here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-611977429
> There appears to be a compilation failure in Java
@reuvenlax fixed.
> There appears to be a compilation failure in Java
@reuvenlax fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614088186
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-611782665
There appears to be a compilation failure in Java
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607102494
Run Direct ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607102824
Run Samza ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612853852
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601218708
Run Direct ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] shehzaadn-vd commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
shehzaadn-vd commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r402698024
##########
File path: examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
##########
@@ -386,7 +385,8 @@ public FormatForDatastore(String kind, String ancestorKey) {
@ProcessElement
public void processElement(ProcessContext c) {
Entity.Builder entityBuilder = Entity.newBuilder();
- Key key = makeKey(makeKey(kind, ancestorKey).build(), kind, c.element().getKey()).build();
+ com.google.datastore.v1.Key key =
+ makeKey(makeKey(kind, ancestorKey).build(), kind, c.element().getKey()).build();
Review comment:
the changes in this file don't seem necessary. We've got rid of an import but introduced a fully qualified class name for the same import instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev removed a comment on issue #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601197747
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612852796
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612852841
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612853075
Run Dataflow Validatesrunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614746001
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607941209
@reuvenlax PTAL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607102684
Run Gearpump ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614087431
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607100998
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r396596657
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvoker.java
##########
@@ -167,6 +167,8 @@ void invokeOnTimer(
/** Provide a reference to the input element. */
InputT element(DoFn<InputT, OutputT> doFn);
+ Object key();
Review comment:
Need javadoc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r397529060
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
##########
@@ -847,6 +847,7 @@ protected void fireTimer(InternalTimer<ByteBuffer, TimerData> timer) {
pushbackDoFnRunner.onTimer(
timerData.getTimerId(),
timerData.getTimerFamilyId(),
+ "",
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali removed a comment on issue #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612835446
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612853924
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r410196509
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
##########
@@ -837,11 +837,15 @@ public void onProcessingTime(InternalTimer<ByteBuffer, TimerData> timer) {
// allow overriding this in ExecutableStageDoFnOperator to set the key context
protected void fireTimerInternal(Object key, TimerData timerData) {
- fireTimer(timerData);
+ if (key instanceof java.nio.ByteBuffer) {
+ key = FlinkKeyUtils.decodeKey((ByteBuffer) key, keyCoder);
Review comment:
The key is always stored as ByteBuffer. This is also reflected in the method calls to this method. Please change the signature of the method to be
```
void fireTimerInternal(ByteBuffer key, TimerData timerdata)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612853780
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev removed a comment on issue #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601199144
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601208276
Run Apex ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r395633155
##########
File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java
##########
@@ -361,6 +361,7 @@ private void processUserTimer(TimerData timer) throws Exception {
fnRunner.onTimer(
timer.getTimerId(),
timer.getTimerFamilyId(),
+ this.stepContext.stateInternals().getKey(),
Review comment:
you'll need the change to StreamingDataflowWorker as well, without it, the key will be null.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601200096
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r403097311
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/BufferedElements.java
##########
@@ -90,7 +90,8 @@ public int hashCode() {
@Override
public void processWith(DoFnRunner doFnRunner) {
- doFnRunner.onTimer(timerId, timerFamilyId, window, timestamp, outputTimestamp, timeDomain);
Review comment:
@reuvenlax done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev removed a comment on issue #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601199519
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612852841
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614087838
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r396597515
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
##########
@@ -1282,6 +1283,14 @@ private static Parameter analyzeExtraParameter(
rawType.equals(Instant.class),
"@Timestamp argument must have type org.joda.time.Instant.");
return Parameter.timestampParameter();
+ } else if (hasAnnotation(DoFn.KeyId.class, param.getAnnotations())) {
+ Type keyType = ((ParameterizedType) inputT.getType()).getActualTypeArguments()[0];
Review comment:
I think we also need to verify that the input type is a KV
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r403098305
##########
File path: examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
##########
@@ -386,7 +385,8 @@ public FormatForDatastore(String kind, String ancestorKey) {
@ProcessElement
public void processElement(ProcessContext c) {
Entity.Builder entityBuilder = Entity.newBuilder();
- Key key = makeKey(makeKey(kind, ancestorKey).build(), kind, c.element().getKey()).build();
+ com.google.datastore.v1.Key key =
+ makeKey(makeKey(kind, ancestorKey).build(), kind, c.element().getKey()).build();
Review comment:
The mentioned Key was conflicting with our DoFn.Key interface.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r396594289
##########
File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
##########
@@ -1155,6 +1156,10 @@ public Instant getStateStartTime() {
// Note that TimerOrElementCoder is a backwards-compatibility class
// that is really a FakeKeyedWorkItemCoder
Coder<?> valueCoder = ((WindowedValueCoder<?>) readCoder).getValueCoder();
+
+ if (valueCoder instanceof KvCoder<?, ?>) {
+ return ((KvCoder<?, ?>) valueCoder).getKeyCoder();
+ }
Review comment:
I'm not sure I understand why this is needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612115813
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-611776987
run dataflow validatesrunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612115861
run dataflow validatesrunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-600679857
Could we ensure that the FnApiDoFnRunner implementation for the key is completed as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601219318
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mwalenia commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
mwalenia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601684734
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612180094
@reuvenlax seems like tests didn't trigger.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612115985
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev removed a comment on issue #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601197814
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614742947
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] shehzaadn-vd commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
shehzaadn-vd commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r402698024
##########
File path: examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
##########
@@ -386,7 +385,8 @@ public FormatForDatastore(String kind, String ancestorKey) {
@ProcessElement
public void processElement(ProcessContext c) {
Entity.Builder entityBuilder = Entity.newBuilder();
- Key key = makeKey(makeKey(kind, ancestorKey).build(), kind, c.element().getKey()).build();
+ com.google.datastore.v1.Key key =
+ makeKey(makeKey(kind, ancestorKey).build(), kind, c.element().getKey()).build();
Review comment:
the changes in this file don't seem necessary. We've got rid of an import but introduced a fully qualified class name instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612853194
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601218578
Run Dataflow ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601243984
seems like this is broken/unsupported in dataflow:
```
java.lang.ClassCastException: java.lang.String cannot be cast to WindowedKey
CombinerDoFn$OnTimerInvoker$t$dA.invokeOnTimer(Unknown Source)
org.apache.beam.sdk.transforms.reflect.ByteBuddyDoFnInvokerFactory$DoFnInvokerBase.invokeOnTimer(ByteBuddyDoFnInvokerFactory.java:229)
org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.onTimer(SimpleDoFnRunner.java:221)
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processUserTimer(SimpleParDoFn.java:373)
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.access$600(SimpleParDoFn.java:74)
org.apache.beam.runners.dataflow.worker.SimpleParDoFn$TimerType$1.processTimer(SimpleParDoFn.java:458)
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processTimers(SimpleParDoFn.java:487)
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processTimers(SimpleParDoFn.java:365)
org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.finish(ParDoOperation.java:52)
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85)
```
(the generic parameter for my key isn't a string either). I'm using a build of the dataflow worker jar from this branch as well, not the one bundled w/ the docker image.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-607102300
Run Apex ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r397529302
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvoker.java
##########
@@ -167,6 +167,8 @@ void invokeOnTimer(
/** Provide a reference to the input element. */
InputT element(DoFn<InputT, OutputT> doFn);
+ Object key();
Review comment:
Added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614087431
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601218474
Run Apex ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r395896091
##########
File path: runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
##########
@@ -686,6 +697,11 @@ public TimeDomain timeDomain() {
return timeDomain;
}
+ @Override
+ public Object key() {
Review comment:
should this instead return KeyT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614087838
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601226883
> Thank you @steveniemitz for catching this bug. Can you please mention how can I reproduce it so that I can fix this.
If you make a DoFn where the key contains a generic parameter you'll run into this. eg:
```
class SomeKey<T> { ... }
class MyDoFn<K> extends DoFn<KV<SomeKey<K>, Object>> { ... }
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r396595875
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
##########
@@ -481,6 +481,12 @@ public Duration getAllowedTimestampSkew() {
String value();
}
+ /** Parameter annotation for the input element key for {@link OnTimer} methods. */
Review comment:
I think should just be named Key, not KeyId.
Also I wouldn't reference OnTimer here, as @Key also makes sense in processElement (instead of getting a KV you should be able to get they key and the value in separate parameters.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601198377
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz edited a comment on issue #11154: [BEAM-1819]
Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
steveniemitz edited a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601243984
seems like this is broken/unsupported in dataflow:
```
java.lang.ClassCastException: java.lang.String cannot be cast to WindowedKey
CombinerDoFn$OnTimerInvoker$t$dA.invokeOnTimer(Unknown Source)
org.apache.beam.sdk.transforms.reflect.ByteBuddyDoFnInvokerFactory$DoFnInvokerBase.invokeOnTimer(ByteBuddyDoFnInvokerFactory.java:229)
org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.SimpleDoFnRunner.onTimer(SimpleDoFnRunner.java:221)
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processUserTimer(SimpleParDoFn.java:373)
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.access$600(SimpleParDoFn.java:74)
org.apache.beam.runners.dataflow.worker.SimpleParDoFn$TimerType$1.processTimer(SimpleParDoFn.java:458)
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processTimers(SimpleParDoFn.java:487)
org.apache.beam.runners.dataflow.worker.SimpleParDoFn.processTimers(SimpleParDoFn.java:365)
org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.finish(ParDoOperation.java:52)
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85)
```
(the generic parameter for my key isn't a string either). I'm using a build of the dataflow worker jar from this branch as well, not the one bundled w/ the docker image.
edit:
ah, I see [here](runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java) you're just passing an empty string in for the key.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601208448
Run Dataflow ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601199717
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601208276
Run Apex ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali edited a comment on issue #11154: [BEAM-1819]
Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali edited a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614042905
@reuvenlax @iemejia could you please run the following tests:
Run Flink ValidatesRunner
Run Java Flink PortableValidatesRunner Streaming
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-600576899
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601219132
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r395896952
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkStatefulDoFnFunction.java
##########
@@ -218,6 +218,7 @@ private void fireTimer(TimerInternals.TimerData timer, DoFnRunner<KV<K, V>, Outp
doFnRunner.onTimer(
timer.getTimerId(),
timer.getTimerFamilyId(),
+ "",
Review comment:
It looks like it should be easy to set the correct key here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-611977812
> There appears to be a compilation failure in Java
@reuvenlax fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612180516
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614088186
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r397529536
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
##########
@@ -1282,6 +1283,14 @@ private static Parameter analyzeExtraParameter(
rawType.equals(Instant.class),
"@Timestamp argument must have type org.joda.time.Instant.");
return Parameter.timestampParameter();
+ } else if (hasAnnotation(DoFn.KeyId.class, param.getAnnotations())) {
+ Type keyType = ((ParameterizedType) inputT.getType()).getActualTypeArguments()[0];
Review comment:
Added with a test case
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-609100632
@steveniemitz in principle, removing a state cell is absolutely a backwards-incompatible change, just like removing a GroupByKey is (for the latter, the Dataflow requires the user to explicitly allow the deletion on update). There are multiple things that can go wrong if a state cell is removed accidentally. In practice, I'm not 100% sure if Dataflow or any other runner currently validates compatibility at the state cell level or not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] reuvenlax commented on issue #11154: [BEAM-1819] Key should
be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612180668
run flink validatesrunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r410224185
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
##########
@@ -837,11 +837,15 @@ public void onProcessingTime(InternalTimer<ByteBuffer, TimerData> timer) {
// allow overriding this in ExecutableStageDoFnOperator to set the key context
protected void fireTimerInternal(Object key, TimerData timerData) {
- fireTimer(timerData);
+ if (key instanceof java.nio.ByteBuffer) {
+ key = FlinkKeyUtils.decodeKey((ByteBuffer) key, keyCoder);
Review comment:
@mxm, upon debugging, I noticed that timer.getKey() returns String, Integer, as well as ByteBuffer. That's why when I changed the method signature, DoFnOperatorTest failed with ClassCastException. And I have added an instanceof check
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612853678
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r397529225
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
##########
@@ -481,6 +481,12 @@ public Duration getAllowedTimestampSkew() {
String value();
}
+ /** Parameter annotation for the input element key for {@link OnTimer} methods. */
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r395644552
##########
File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/SimpleParDoFn.java
##########
@@ -361,6 +361,7 @@ private void processUserTimer(TimerData timer) throws Exception {
fnRunner.onTimer(
timer.getTimerId(),
timer.getTimerFamilyId(),
+ this.stepContext.stateInternals().getKey(),
Review comment:
@steveniemitz added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia commented on issue #11154: [BEAM-1819] Key should be
available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614088075
Run Java Flink PortableValidatesRunner Streaming
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz edited a comment on issue #11154: [BEAM-1819]
Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
steveniemitz edited a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601221088
I think the type check for `@KeyId` breaks if the key is generic. I was trying this out and got:
```
java.lang.IllegalArgumentException: CombinerDoFn, @OnTimer onTriggerTimer(BoundedWindow, WindowedKey, CombiningState, CombiningState, ValueState, CombiningState, BagState, OutputReceiver): @Key argument is expected to be type of WindowedKey<K>, but found class WindowedKey
```
I think changing the check from `keyType.equals(rawType)` to `TypeDescriptor.of(keyType).equals(paramT)` should solve it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-600808846
R: @reuvenlax.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r397529536
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java
##########
@@ -1282,6 +1283,14 @@ private static Parameter analyzeExtraParameter(
rawType.equals(Instant.class),
"@Timestamp argument must have type org.joda.time.Instant.");
return Parameter.timestampParameter();
+ } else if (hasAnnotation(DoFn.KeyId.class, param.getAnnotations())) {
+ Type keyType = ((ParameterizedType) inputT.getType()).getActualTypeArguments()[0];
Review comment:
@reuvenlax , Added with a test case
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] iemejia removed a comment on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
iemejia removed a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-612853600
Run Dataflow Validatesrunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-608516020
I hate to pile on more changes, but what do you think about changing [GroupIntoBatches](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java) to use this? The change should be trivial, and it seems like it'd be good to have a real use-case already using this feature.
I guess this is assuming we have support for this in all major runners.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz edited a comment on issue #11154: [BEAM-1819]
Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
steveniemitz edited a comment on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601226883
> Thank you @steveniemitz for catching this bug. Can you please mention how can I reproduce it so that I can fix this.
If you make a DoFn where the key contains a generic parameter you'll run into this. eg:
```
class SomeKey<T> { ... }
class MyDoFn<K> extends DoFn<KV<SomeKey<K>, Object>> { ... }
```
Just tested with my suggestion, that seems to fix it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] rehmanmuradali commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
rehmanmuradali commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-614042905
@iemejia could you please run the following tests:
Run Flink ValidatesRunner
Run Java Flink PortableValidatesRunner Streaming
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] steveniemitz commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
steveniemitz commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-609101077
Hm, well that's certainly unfortunate then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11154:
[BEAM-1819] Key should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#discussion_r410197549
##########
File path: runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
##########
@@ -837,11 +837,15 @@ public void onProcessingTime(InternalTimer<ByteBuffer, TimerData> timer) {
// allow overriding this in ExecutableStageDoFnOperator to set the key context
protected void fireTimerInternal(Object key, TimerData timerData) {
Review comment:
```suggestion
protected void fireTimerInternal(ByteBuffer key, TimerData timerData) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on issue #11154: [BEAM-1819] Key
should be available in @OnTimer methods
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #11154: [BEAM-1819] Key should be available in @OnTimer methods
URL: https://github.com/apache/beam/pull/11154#issuecomment-601199519
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services