You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/20 03:24:42 UTC
[GitHub] [beam] boyuanzz opened a new pull request #11177: [BEAM-9562] Add
Timer to Elements proto representation.
boyuanzz opened a new pull request #11177: [BEAM-9562] Add Timer to Elements proto representation.
URL: https://github.com/apache/beam/pull/11177
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on a change in pull request #11177:
[BEAM-9562] Add Timer to Elements proto representation.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #11177: [BEAM-9562] Add Timer to Elements proto representation.
URL: https://github.com/apache/beam/pull/11177#discussion_r395426138
##########
File path: model/fn-execution/src/main/proto/beam_fn_api.proto
##########
@@ -533,8 +538,33 @@ message Elements {
bytes data = 3;
}
+ // Represent the encoded user timer for a given instruction, transform and
+ // timer id.
+ message Timer {
+ // (Required) A reference to an active instruction request with the given
+ // instruction id.
+ string instruction_id = 1;
Review comment:
```suggestion
string instruction_id = 1;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on a change in pull request #11177:
[BEAM-9562] Add Timer to Elements proto representation.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #11177: [BEAM-9562] Add Timer to Elements proto representation.
URL: https://github.com/apache/beam/pull/11177#discussion_r395426161
##########
File path: model/fn-execution/src/main/proto/beam_fn_api.proto
##########
@@ -533,8 +538,33 @@ message Elements {
bytes data = 3;
}
+ // Represent the encoded user timer for a given instruction, transform and
+ // timer id.
+ message Timer {
+ // (Required) A reference to an active instruction request with the given
+ // instruction id.
+ string instruction_id = 1;
+ // (Required) A definition representing a consumer or producer of this data.
+ // If received by a harness, this represents the consumer within that
+ // harness that should consume these bytes. If sent by a harness, this
+ // represents the producer of these bytes.
+ string transform_id = 2;
Review comment:
```suggestion
string transform_id = 2;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] boyuanzz commented on issue #11177: [BEAM-9562] Add Timer to
Elements proto representation.
Posted by GitBox <gi...@apache.org>.
boyuanzz commented on issue #11177: [BEAM-9562] Add Timer to Elements proto representation.
URL: https://github.com/apache/beam/pull/11177#issuecomment-601853060
Thanks for your review!
Java PreCommit failures are not related. I'm going to merge it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on a change in pull request #11177:
[BEAM-9562] Add Timer to Elements proto representation.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #11177: [BEAM-9562] Add Timer to Elements proto representation.
URL: https://github.com/apache/beam/pull/11177#discussion_r395426588
##########
File path: model/fn-execution/src/main/proto/beam_fn_api.proto
##########
@@ -533,8 +538,33 @@ message Elements {
bytes data = 3;
}
+ // Represent the encoded user timer for a given instruction, transform and
+ // timer id.
+ message Timer {
+ // (Required) A reference to an active instruction request with the given
+ // instruction id.
+ string instruction_id = 1;
+ // (Required) A definition representing a consumer or producer of this data.
+ // If received by a harness, this represents the consumer within that
+ // harness that should consume these bytes. If sent by a harness, this
+ // represents the producer of these bytes.
+ string transform_id = 2;
+ // (Optional) The local timer name which can be mapped to
+ // timer specification.
Review comment:
```suggestion
// (Optional) The local timer name used to identify the associated timer specification.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] boyuanzz merged pull request #11177: [BEAM-9562] Add Timer
to Elements proto representation.
Posted by GitBox <gi...@apache.org>.
boyuanzz merged pull request #11177: [BEAM-9562] Add Timer to Elements proto representation.
URL: https://github.com/apache/beam/pull/11177
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on a change in pull request #11177:
[BEAM-9562] Add Timer to Elements proto representation.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #11177: [BEAM-9562] Add Timer to Elements proto representation.
URL: https://github.com/apache/beam/pull/11177#discussion_r395426182
##########
File path: model/fn-execution/src/main/proto/beam_fn_api.proto
##########
@@ -533,8 +538,33 @@ message Elements {
bytes data = 3;
}
+ // Represent the encoded user timer for a given instruction, transform and
+ // timer id.
+ message Timer {
+ // (Required) A reference to an active instruction request with the given
+ // instruction id.
+ string instruction_id = 1;
+ // (Required) A definition representing a consumer or producer of this data.
+ // If received by a harness, this represents the consumer within that
+ // harness that should consume these bytes. If sent by a harness, this
+ // represents the producer of these bytes.
+ string transform_id = 2;
+ // (Optional) The local timer name which can be mapped to
+ // timer specification.
+ string timer_id = 3;
Review comment:
```suggestion
string timer_id = 3;
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on a change in pull request #11177:
[BEAM-9562] Add Timer to Elements proto representation.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #11177: [BEAM-9562] Add Timer to Elements proto representation.
URL: https://github.com/apache/beam/pull/11177#discussion_r395426028
##########
File path: model/fn-execution/src/main/proto/beam_fn_api.proto
##########
@@ -164,6 +164,11 @@ message ProcessBundleDescriptor {
// data plane or if any of the transforms rely on user state or side inputs.
org.apache.beam.model.pipeline.v1.ApiServiceDescriptor
state_api_service_descriptor = 7;
+
+ // A descriptor describing the end point to use for Data API to use user
+ // timers.
Review comment:
```suggestion
// A descriptor describing the end point to use for Data API for user timers.
// Required if the ProcessBundleDescriptor contains any transforms that have user timers.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on a change in pull request #11177:
[BEAM-9562] Add Timer to Elements proto representation.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #11177: [BEAM-9562] Add Timer to Elements proto representation.
URL: https://github.com/apache/beam/pull/11177#discussion_r395426658
##########
File path: model/fn-execution/src/main/proto/beam_fn_api.proto
##########
@@ -533,8 +538,33 @@ message Elements {
bytes data = 3;
}
+ // Represent the encoded user timer for a given instruction, transform and
+ // timer id.
+ message Timer {
+ // (Required) A reference to an active instruction request with the given
+ // instruction id.
+ string instruction_id = 1;
+ // (Required) A definition representing a consumer or producer of this data.
+ // If received by a harness, this represents the consumer within that
+ // harness that should consume these bytes. If sent by a harness, this
+ // represents the producer of these bytes.
+ string transform_id = 2;
+ // (Optional) The local timer name which can be mapped to
+ // timer specification.
+ string timer_id = 3;
+ // (Optional) Represents a logical byte stream of a timer. Encoded according
+ // to the coder in the timer spec.
+ // An empty data block represents the end of stream for the given
+ // instruction and transform.
+ bytes timer = 4;
+ }
+
+
// (Required) A list containing parts of logical byte streams.
Review comment:
```suggestion
// (Optional) A list containing parts of logical byte streams.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on a change in pull request #11177:
[BEAM-9562] Add Timer to Elements proto representation.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #11177: [BEAM-9562] Add Timer to Elements proto representation.
URL: https://github.com/apache/beam/pull/11177#discussion_r395426301
##########
File path: model/fn-execution/src/main/proto/beam_fn_api.proto
##########
@@ -533,8 +538,33 @@ message Elements {
bytes data = 3;
}
+ // Represent the encoded user timer for a given instruction, transform and
+ // timer id.
+ message Timer {
+ // (Required) A reference to an active instruction request with the given
+ // instruction id.
+ string instruction_id = 1;
+ // (Required) A definition representing a consumer or producer of this data.
+ // If received by a harness, this represents the consumer within that
+ // harness that should consume these bytes. If sent by a harness, this
+ // represents the producer of these bytes.
Review comment:
```suggestion
// harness that should consume these timers. If sent by a harness, this
// represents the producer of these timers.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services