You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/05/03 10:40:13 UTC
[GitHub] [beam] mxm opened a new pull request #11597: [BEAM-9874] Support clearing timers in portable batch mode (Spark/Flink)
mxm opened a new pull request #11597:
URL: https://github.com/apache/beam/pull/11597
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on pull request #11597: [BEAM-9874] Support clearing timers in portable batch mode (Spark/Flink)
Posted by GitBox <gi...@apache.org>.
mxm commented on pull request #11597:
URL: https://github.com/apache/beam/pull/11597#issuecomment-623156577
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #11597: [BEAM-9874] Support clearing timers in portable batch mode (Spark/Flink)
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #11597:
URL: https://github.com/apache/beam/pull/11597#discussion_r419607989
##########
File path: runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
##########
@@ -167,20 +165,17 @@ public void deleteTimer(StateNamespace namespace, String timerId, TimeDomain tim
@Deprecated
@Override
public void deleteTimer(StateNamespace namespace, String timerId, String timerFamilyId) {
- TimerData existing = existingTimers.get(namespace, timerId + '+' + timerFamilyId);
- if (existing != null) {
- deleteTimer(existing);
+ TimerData removedTimer = existingTimers.remove(namespace, timerId + '+' + timerFamilyId);
+ if (removedTimer != null) {
+ timersForDomain(removedTimer.getDomain()).remove(removedTimer);
}
}
/** @deprecated use {@link #deleteTimer(StateNamespace, String, TimeDomain)}. */
Review comment:
It seems strange that the deprecated method is the one being used, and the recommended method is not implemented.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ibzib commented on a change in pull request #11597: [BEAM-9874] Support clearing timers in portable batch mode (Spark/Flink)
Posted by GitBox <gi...@apache.org>.
ibzib commented on a change in pull request #11597:
URL: https://github.com/apache/beam/pull/11597#discussion_r419658069
##########
File path: runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
##########
@@ -167,20 +165,17 @@ public void deleteTimer(StateNamespace namespace, String timerId, TimeDomain tim
@Deprecated
@Override
public void deleteTimer(StateNamespace namespace, String timerId, String timerFamilyId) {
- TimerData existing = existingTimers.get(namespace, timerId + '+' + timerFamilyId);
- if (existing != null) {
- deleteTimer(existing);
+ TimerData removedTimer = existingTimers.remove(namespace, timerId + '+' + timerFamilyId);
+ if (removedTimer != null) {
+ timersForDomain(removedTimer.getDomain()).remove(removedTimer);
}
}
/** @deprecated use {@link #deleteTimer(StateNamespace, String, TimeDomain)}. */
Review comment:
Maybe worth a JIRA/refactor? I'll leave it up to you since I am not as familiar with this code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on pull request #11597: [BEAM-9874] Support clearing timers in portable batch mode (Spark/Flink)
Posted by GitBox <gi...@apache.org>.
mxm commented on pull request #11597:
URL: https://github.com/apache/beam/pull/11597#issuecomment-623091663
Run Python Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on pull request #11597: [BEAM-9874] Support clearing timers in portable batch mode (Spark/Flink)
Posted by GitBox <gi...@apache.org>.
mxm commented on pull request #11597:
URL: https://github.com/apache/beam/pull/11597#issuecomment-623340277
I'm merging to unblock other PRs. I'm happy to follow-up though with regards to any comments you might have.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on a change in pull request #11597: [BEAM-9874] Support clearing timers in portable batch mode (Spark/Flink)
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11597:
URL: https://github.com/apache/beam/pull/11597#discussion_r419134807
##########
File path: runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
##########
@@ -167,20 +165,17 @@ public void deleteTimer(StateNamespace namespace, String timerId, TimeDomain tim
@Deprecated
@Override
public void deleteTimer(StateNamespace namespace, String timerId, String timerFamilyId) {
- TimerData existing = existingTimers.get(namespace, timerId + '+' + timerFamilyId);
- if (existing != null) {
- deleteTimer(existing);
+ TimerData removedTimer = existingTimers.remove(namespace, timerId + '+' + timerFamilyId);
+ if (removedTimer != null) {
+ timersForDomain(removedTimer.getDomain()).remove(removedTimer);
}
}
/** @deprecated use {@link #deleteTimer(StateNamespace, String, TimeDomain)}. */
@Deprecated
@Override
public void deleteTimer(TimerData timer) {
- WindowTracing.trace("{}.deleteTimer: {}", getClass().getSimpleName(), timer);
- existingTimers.remove(
- timer.getNamespace(), timer.getTimerId() + '+' + timer.getTimerFamilyId());
- timersForDomain(timer.getDomain()).remove(timer);
Review comment:
There was nasty bug here which prevented timer deletion because the to-be-deleted TimerData had a different timestamp than the original timer, leading to not actually deleting the timer from the map.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on a change in pull request #11597: [BEAM-9874] Support clearing timers in portable batch mode (Spark/Flink)
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11597:
URL: https://github.com/apache/beam/pull/11597#discussion_r419632632
##########
File path: runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
##########
@@ -167,20 +165,17 @@ public void deleteTimer(StateNamespace namespace, String timerId, TimeDomain tim
@Deprecated
@Override
public void deleteTimer(StateNamespace namespace, String timerId, String timerFamilyId) {
- TimerData existing = existingTimers.get(namespace, timerId + '+' + timerFamilyId);
- if (existing != null) {
- deleteTimer(existing);
+ TimerData removedTimer = existingTimers.remove(namespace, timerId + '+' + timerFamilyId);
+ if (removedTimer != null) {
+ timersForDomain(removedTimer.getDomain()).remove(removedTimer);
}
}
/** @deprecated use {@link #deleteTimer(StateNamespace, String, TimeDomain)}. */
Review comment:
yeah, actually that might have been a mistake when the timer family was added because the old way was to use `TimerData` and the new way is to use timerId/timerFamily.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org