You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/19 00:08:36 UTC
[GitHub] [beam] lukecwik opened a new pull request #11167: [BEAM-4374]
Define the protos for a "short" id mechanism for metrics
lukecwik opened a new pull request #11167: [BEAM-4374] Define the protos for a "short" id mechanism for metrics
URL: https://github.com/apache/beam/pull/11167
Typical flow:
Runner requests metrics and SDK responds with metrics data and ids, SDK stores id -> full MonitoringInfo mapping.
if Runner is missing ids:
Runner requests full data given for missing ids and SDK provides full data
Runner "remembers" id -> full mapping for future process bundle responses
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11167: [BEAM-4374] Define the
protos for a "short" id mechanism for metrics
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11167: [BEAM-4374] Define the protos for a "short" id mechanism for metrics
URL: https://github.com/apache/beam/pull/11167#issuecomment-601509826
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11167: [BEAM-4374] Define the
protos for a "short" id mechanism for metrics
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11167: [BEAM-4374] Define the protos for a "short" id mechanism for metrics
URL: https://github.com/apache/beam/pull/11167#issuecomment-600919341
R: @robertwb
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11167: [BEAM-4374] Define the
protos for a "short" id mechanism for metrics
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11167: [BEAM-4374] Define the protos for a "short" id mechanism for metrics
URL: https://github.com/apache/beam/pull/11167#issuecomment-601478882
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11167: [BEAM-4374] Define the
protos for a "short" id mechanism for metrics
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11167: [BEAM-4374] Define the protos for a "short" id mechanism for metrics
URL: https://github.com/apache/beam/pull/11167#issuecomment-601256422
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] robertwb commented on a change in pull request #11167:
[BEAM-4374] Define the protos for a "short" id mechanism for metrics
Posted by GitBox <gi...@apache.org>.
robertwb commented on a change in pull request #11167: [BEAM-4374] Define the protos for a "short" id mechanism for metrics
URL: https://github.com/apache/beam/pull/11167#discussion_r395215550
##########
File path: model/fn-execution/src/main/proto/beam_fn_api.proto
##########
@@ -277,6 +291,21 @@ message ProcessBundleProgressRequest {
string instruction_id = 1;
}
+// A request to provide full MonitoringInfo for a given bundle.
+//
+// An SDK can report metrics using an identifier that only contains the
+// associated payload. A runner who wants to receive the full metrics
+// information can request all the monitoring metadata via a
+// ProcessBundleProgressMetadataRequest providing a list of ids as necessary.
+//
+// The SDK is allowed to reuse the identifiers across multiple bundles as long
+// as the MonitoringInfo could be reconstructed fully by overwriting its
+// payload field with the bytes specified here.
+message ProcessBundleProgressMetadataRequest {
+ // A list of ids for which the full MonitoringInfo is requested for.
+ repeated string id = 1;
Review comment:
Maybe call this `monitioring_info_id` to distinguish this from the many other ids (like instruction id, process bundle id, etc.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11167: [BEAM-4374] Define the
protos for a "short" id mechanism for metrics
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11167: [BEAM-4374] Define the protos for a "short" id mechanism for metrics
URL: https://github.com/apache/beam/pull/11167#issuecomment-601478970
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11167: [BEAM-4374] Define the
protos for a "short" id mechanism for metrics
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11167: [BEAM-4374] Define the protos for a "short" id mechanism for metrics
URL: https://github.com/apache/beam/pull/11167#issuecomment-601291830
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on a change in pull request #11167:
[BEAM-4374] Define the protos for a "short" id mechanism for metrics
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #11167: [BEAM-4374] Define the protos for a "short" id mechanism for metrics
URL: https://github.com/apache/beam/pull/11167#discussion_r395299472
##########
File path: model/fn-execution/src/main/proto/beam_fn_api.proto
##########
@@ -277,6 +291,21 @@ message ProcessBundleProgressRequest {
string instruction_id = 1;
}
+// A request to provide full MonitoringInfo for a given bundle.
+//
+// An SDK can report metrics using an identifier that only contains the
+// associated payload. A runner who wants to receive the full metrics
+// information can request all the monitoring metadata via a
+// ProcessBundleProgressMetadataRequest providing a list of ids as necessary.
+//
+// The SDK is allowed to reuse the identifiers across multiple bundles as long
+// as the MonitoringInfo could be reconstructed fully by overwriting its
+// payload field with the bytes specified here.
+message ProcessBundleProgressMetadataRequest {
+ // A list of ids for which the full MonitoringInfo is requested for.
+ repeated string id = 1;
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11167: [BEAM-4374] Define the
protos for a "short" id mechanism for metrics
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11167: [BEAM-4374] Define the protos for a "short" id mechanism for metrics
URL: https://github.com/apache/beam/pull/11167#issuecomment-601256530
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik merged pull request #11167: [BEAM-4374] Define the
protos for a "short" id mechanism for metrics
Posted by GitBox <gi...@apache.org>.
lukecwik merged pull request #11167: [BEAM-4374] Define the protos for a "short" id mechanism for metrics
URL: https://github.com/apache/beam/pull/11167
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] lukecwik commented on issue #11167: [BEAM-4374] Define the
protos for a "short" id mechanism for metrics
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #11167: [BEAM-4374] Define the protos for a "short" id mechanism for metrics
URL: https://github.com/apache/beam/pull/11167#issuecomment-601509805
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services