You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/06/26 00:29:05 UTC
[GitHub] [beam] youngoli opened a new pull request #12095: [BEAM-10311] Sharing restriction trackers between SDF and DataSource.
youngoli opened a new pull request #12095:
URL: https://github.com/apache/beam/pull/12095
Adding a channel that can be used to retrieve the current restriction
tracker from the DataSource.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark
--- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #12095: [BEAM-10311] Sharing restriction trackers between SDF and DataSource.
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #12095:
URL: https://github.com/apache/beam/pull/12095#discussion_r446327844
##########
File path: sdks/go/pkg/beam/core/runtime/exec/sdf.go
##########
@@ -217,6 +218,11 @@ type ProcessSizedElementsAndRestrictions struct {
PDo *ParDo
inv *ctInvoker
+
+ // rt allows this unit to send out restriction trackers being processed.
Review comment:
Typo" Rt
```suggestion
// Rt allows this unit to send out restriction trackers being processed.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on a change in pull request #12095: [BEAM-10311] Sharing restriction trackers between SDF and DataSource.
Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #12095:
URL: https://github.com/apache/beam/pull/12095#discussion_r446445983
##########
File path: sdks/go/pkg/beam/core/runtime/exec/sdf.go
##########
@@ -217,6 +218,11 @@ type ProcessSizedElementsAndRestrictions struct {
PDo *ParDo
inv *ctInvoker
+
+ // rt allows this unit to send out restriction trackers being processed.
Review comment:
Fixed. (Also switched the channel from `*RTracker` to `RTracker`)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #12095: [BEAM-10311] Sharing restriction trackers between SDF and DataSource.
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #12095:
URL: https://github.com/apache/beam/pull/12095#discussion_r446446107
##########
File path: sdks/go/pkg/beam/core/runtime/exec/sdf.go
##########
@@ -217,6 +218,11 @@ type ProcessSizedElementsAndRestrictions struct {
PDo *ParDo
inv *ctInvoker
+
+ // rt allows this unit to send out restriction trackers being processed.
Review comment:
Ah yes. Good catch. Don't use pointers to interface types :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli merged pull request #12095: [BEAM-10311] Sharing restriction trackers between SDF and DataSource.
Posted by GitBox <gi...@apache.org>.
youngoli merged pull request #12095:
URL: https://github.com/apache/beam/pull/12095
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on pull request #12095: [BEAM-10311] Sharing restriction trackers between SDF and DataSource.
Posted by GitBox <gi...@apache.org>.
youngoli commented on pull request #12095:
URL: https://github.com/apache/beam/pull/12095#issuecomment-649885158
R: @lostluck
Ended up being a really small PR. I tried to think of a way to test it (like testing that the channel gets set in the DataSource maybe), but it seems too specific of an interaction to test, and any test for it would be a bit complicated.
For context of what I'm intending when it comes to the actual splitting: The idea is the DataSource will receive the restriction tracker from the channel when it needs to check progress or perform a split, and then return the restriction tracker to the channel when done with it. The goal is to make sure there's only one copy of that tracker being sent around, so that nothing else is using the tracker when processing is done and the tracker is getting deleted.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org