You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 12:32:43 UTC
[GitHub] [beam] damccorm opened a new issue, #19632: Update throttling counters to use generic throttling-msecs metric and dedicated namespace
damccorm opened a new issue, #19632:
URL: https://github.com/apache/beam/issues/19632
Update throttling counters to use generic throttling-msecs metric and dedicated namespace. Currently, throttle time metric is saved in a user counter that a namespace is a class name and a counter name is "cumulativeThrottlingSeconds".
Imported from Jira [BEAM-7863](https://issues.apache.org/jira/browse/BEAM-7863). Original Jira may contain additional context.
Reported by: heejong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Naireen commented on issue #19632: Update throttling counters to use generic throttling-msecs metric and dedicated namespace
Posted by "Naireen (via GitHub)" <gi...@apache.org>.
Naireen commented on issue #19632:
URL: https://github.com/apache/beam/issues/19632#issuecomment-1561992248
There are some throttling counters that don't follow that convention, eg the BQ streaming inserts throttling counter: https://github.com/apache/beam/blob/d91ea8e44918a7ef10ac8b33001cddab61bcedc5/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#LL561C1-L562C71
How do you envision the end state to look? If there is a single metric for the throttling counter, should all IO's update only that counter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #19632: Update throttling counters to use generic throttling-msecs metric and dedicated namespace
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #19632:
URL: https://github.com/apache/beam/issues/19632#issuecomment-1562893582
@ihji filed this one originally, I don't have a ton of context. @johnjcasey may have thoughts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on issue #19632: Update throttling counters to use generic throttling-msecs metric and dedicated namespace
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on issue #19632:
URL: https://github.com/apache/beam/issues/19632#issuecomment-1650381180
Dataflow side just discussed this a bit, so forwarding some key points:
Beam Metrics over the FnAPI are always qualified by Transform, Namespace, Name, and additionally by Bundle. That last one is ephemeral, and used to get precise counters WRT bundle retries.
So in that respect, a runner can already always "know" which transform a counter comes from, and then act accordingly. But for the most part, there's no reason to distinguish between transforms that are throttling. The key part then is deciding officially on a common "beam" namespace (I'd suggest `beam` or `beam:standard:counter`, over something javalike `org.apache.beam.standard.count`) (I'll go with `beam` for now) and a name for the counter itself, which might as well be `throttling-msecs` since there are several IOs already doing that.
So the hard part of this (aside from a dev list community discussion of the proposal) would be clearly documenting that the `beam` namespace has certain reserved counter names, that runners may take special note of for whatever they need. Throttling is one case here. But even things like `rpc-count`, `rpc-retries`, etc etc... And this should be documented clearly in the "IO" guide, and any SDK building guide on the beam website.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #19632: Update throttling counters to use generic throttling-msecs metric and dedicated namespace
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey commented on issue #19632:
URL: https://github.com/apache/beam/issues/19632#issuecomment-1564440444
I don't have context on this. This looks like something that requires a bit of a doc to find the vision
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org