You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/15 17:52:50 UTC

[GitHub] [beam] arne-alex opened a new issue, #23262: [Feature Request]: Extend Google Cloud Dataflow runner with per-key latency metrics

arne-alex opened a new issue, #23262:
URL: https://github.com/apache/beam/issues/23262

   ### What would you like to happen?
   
   We'll extend Google Cloud Dataflow runners with new latency metrics. The new metrics will track time spent in various processing states in the runner. In particular, they allow to distinguish between time spent running pipeline code and time spent waiting for Google Cloud Dataflow backends.
   
   Planned work:
   1. extend the Windmill protocol with new fields in KeyedGetDataRequest, defined at https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/windmill/src/main/proto/windmill.proto
   2. add another state that represents waiting for further data here: https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java#L1169
   3. populate the new fields from google-cloud-dataflow-runner code here: https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java#L2378
   
   
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: runner-dataflow


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] arne-alex closed issue #23262: [Feature Request]: Extend Google Cloud Dataflow runner with per-key latency metrics

Posted by GitBox <gi...@apache.org>.
arne-alex closed issue #23262: [Feature Request]: Extend Google Cloud Dataflow runner with per-key latency metrics
URL: https://github.com/apache/beam/issues/23262


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] arne-alex commented on issue #23262: [Feature Request]: Extend Google Cloud Dataflow runner with per-key latency metrics

Posted by GitBox <gi...@apache.org>.
arne-alex commented on issue #23262:
URL: https://github.com/apache/beam/issues/23262#issuecomment-1248421546

   .take-issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org