You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "lostluck (via GitHub)" <gi...@apache.org> on 2024/07/17 22:44:12 UTC
[I] [prism] Support Attempted Metrics [beam]
lostluck opened a new issue, #31926:
URL: https://github.com/apache/beam/issues/31926
Prism doesn't currently populate Attempted metrics, just committed metrics, when returning metrics through Job Logs.
In principle, for completed jobs, the values should be identical. They should only be different while a Job is executing.
The short version: Committed metrics should be all metrics from ProcessBundleResponses, while tentative metrics are those *plus* anything from a ProcessBundleProgressResponse.
Metrics are accumulated here.
https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/prism/internal/jobservices/metrics.go#L483
This would allow the following Java Validates Runner tests to pass:
MetricsTest$AttemptedMetricTests.testAttemptedDistributionMetrics
MetricsTest$AttemptedMetricTests.testAttemptedCounterMetrics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [prism] Support Attempted Metrics [beam]
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on issue #31926:
URL: https://github.com/apache/beam/issues/31926#issuecomment-2291575394
It looks like the root issue here is that the Loopback worker for Java doesn't propagate Runner Capabilities. This was surprising, but also not, as Java doesn't make use of loopback mode as much of the other SDKs. The ExternalWorker service needs to call the provision endpoint when available and override whatever it thinks it needs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [prism] Support Attempted Metrics [beam]
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck closed issue #31926: [prism] Support Attempted Metrics
URL: https://github.com/apache/beam/issues/31926
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org