You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/05/08 15:09:02 UTC
[GitHub] [beam] epicfaace opened a new pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)
epicfaace opened a new pull request #11642:
URL: https://github.com/apache/beam/pull/11642
**Please** add a meaningful description for your change here
All calls to .checkpoint() need to be replaced by .try_claim(), and the .checkpoint() function was removed from OffsetRestrictionTracker in this commit (https://github.com/apache/beam/commit/b94dca2b1df4a3aea66d822299a58c97accc0541#diff-e642b3261abdcbd9862e7e3978359e76L170).
Because the DirectRunner was still calling .checkpoint(), it ended up crashing at times when trying to run using the OffsetRestrictionTracker. This PR fixes it.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [x] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)
Posted by GitBox <gi...@apache.org>.
epicfaace commented on pull request #11642:
URL: https://github.com/apache/beam/pull/11642#issuecomment-744394992
Still need to take a look at this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)
Posted by GitBox <gi...@apache.org>.
epicfaace commented on pull request #11642:
URL: https://github.com/apache/beam/pull/11642#issuecomment-665635761
Still need to take a look at this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11642:
URL: https://github.com/apache/beam/pull/11642#issuecomment-1032573561
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] stale[bot] commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)
Posted by GitBox <gi...@apache.org>.
stale[bot] commented on pull request #11642:
URL: https://github.com/apache/beam/pull/11642#issuecomment-743934671
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)
Posted by GitBox <gi...@apache.org>.
epicfaace commented on pull request #11642:
URL: https://github.com/apache/beam/pull/11642#issuecomment-1032676610
not stale
--
Ashwin Ramaswami
On Tue, Feb 8, 2022 at 7:50 AM github-actions[bot] ***@***.***>
wrote:
> This pull request has been marked as stale due to 60 days of inactivity.
> It will be closed in 1 week if no further activity occurs. If you think
> that’s incorrect or this pull request requires a review, please simply
> write any comment. If closed, you can revive the PR at any time and
> @mention a reviewer or discuss it on the ***@***.*** list. Thank
> you for your contributions.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/11642#issuecomment-1032573561>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAM4MX5OZTIDTENF3IDEGVLU2EGQZANCNFSM4M4HWU2A>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #11642:
URL: https://github.com/apache/beam/pull/11642#issuecomment-625872963
R: @boyuanzz When the PR is ready.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] stale[bot] commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)
Posted by GitBox <gi...@apache.org>.
stale[bot] commented on pull request #11642:
URL: https://github.com/apache/beam/pull/11642#issuecomment-665427361
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)
Posted by GitBox <gi...@apache.org>.
epicfaace commented on pull request #11642:
URL: https://github.com/apache/beam/pull/11642#issuecomment-704219078
Still need to take a look at this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11642:
URL: https://github.com/apache/beam/pull/11642#issuecomment-635705379
> > This is a single line change and passing all the tests. If the change make sense can we merge it? (question to @boyuanzz )
>
> I don't think the change is correct. I can open a PR and cc the author.
OK. In that case, no action is required from you. We can wait until @epicfaace updates this. Thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #11642:
URL: https://github.com/apache/beam/pull/11642#issuecomment-635680691
> This is a single line change and passing all the tests. If the change make sense can we merge it? (question to @boyuanzz )
I don't think the change is correct. I can open a PR and cc the author.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on a change in pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on a change in pull request #11642:
URL: https://github.com/apache/beam/pull/11642#discussion_r432230809
##########
File path: sdks/python/apache_beam/runners/direct/sdf_direct_runner.py
##########
@@ -464,7 +464,7 @@ def initiate_checkpoint():
with self._checkpoint_lock:
if checkpoint_state.checkpointed:
return
- checkpoint_state.residual_restriction = tracker.checkpoint()
+ checkpoint_state.residual_restriction = tracker.try_claim(0)
Review comment:
Are you meaning `try_split(0)`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] boyuanzz commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on pull request #11642:
URL: https://github.com/apache/beam/pull/11642#issuecomment-631136611
Hi Ashwin, do you want me to start to review now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11642:
URL: https://github.com/apache/beam/pull/11642#issuecomment-635675425
This is a single line change and passing all the tests. If the change make sense can we merge it? (question to @boyuanzz )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] stale[bot] commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)
Posted by GitBox <gi...@apache.org>.
stale[bot] commented on pull request #11642:
URL: https://github.com/apache/beam/pull/11642#issuecomment-703191311
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org