You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "dmateusp (via GitHub)" <gi...@apache.org> on 2023/05/04 15:44:03 UTC

[GitHub] [beam] dmateusp opened a new issue, #24169: [Feature Request]: BigQuery Storage API read throttle

dmateusp opened a new issue, #24169:
URL: https://github.com/apache/beam/issues/24169

   ### What would you like to happen?
   
   Beam version: `2.36.0`
   
   Jobs reading from the BigQuery Storage API can hit Quota limits and eventually fail with `The worker lost contact with the service.` (Dataflow runner)
   
   The reason for the failure can be diagnosed on the API Monitoring (GCP) for BigQuery Storage API calls, where a spike in `429`s can be seen for `ReadRows` when the throughput of the job starts going downhill.
   
   On the GCP side we've tried to resolve it through Quota increases, but it would be helpful to de-risk jobs crashing
   
   related to:
   * https://github.com/apache/beam/pull/15445 <- Shouldn't this PR have pervented this kind of issue?
   
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: io-java-gcp


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] kennknowles commented on issue #24169: [Feature Request]: BigQuery Storage API read throttle

Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #24169:
URL: https://github.com/apache/beam/issues/24169#issuecomment-1335573386

   Sounds good. Thanks for replying!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] dmateusp commented on issue #24169: [Feature Request]: BigQuery Storage API read throttle

Posted by GitBox <gi...@apache.org>.
dmateusp commented on issue #24169:
URL: https://github.com/apache/beam/issues/24169#issuecomment-1335413716

   hey thanks for having a look, it seems like the version of Beam we were using did not include https://github.com/apache/beam/pull/15445
   
   We use Beam through SCIO and we don't have a release that includes Beam `2.43.0` yet, so I wasn't able to test the fix
   
   We could close this issue for now and I could re-open it if we encounter it in 2.43.0 (which I don't think we will, given the fix above)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] kennknowles commented on issue #24169: [Feature Request]: BigQuery Storage API read throttle

Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #24169:
URL: https://github.com/apache/beam/issues/24169#issuecomment-1335389811

   @johnjcasey @reuvenlax @chamikaramj any thoughts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] chamikaramj commented on issue #24169: [Feature Request]: BigQuery Storage API read throttle

Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on issue #24169:
URL: https://github.com/apache/beam/issues/24169#issuecomment-1535072535

   Did you mean https://github.com/apache/beam/pull/15445 (different author) ?
   
   I don't think that changes the retry behavior. Seems like it bumps up the metric used by Dataflow autoscaler to keep track of time the worker is throttled so that job does not unnecessarily scale up in that scenario.
   
   Also note that this is BQ read API (different from BQ Storage Write API) in case that is what you are trying to track here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] liferoad commented on issue #24169: [Feature Request]: BigQuery Storage API read throttle

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #24169:
URL: https://github.com/apache/beam/issues/24169#issuecomment-1535007491

   .reopen-issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] liferoad commented on issue #24169: [Feature Request]: BigQuery Storage API read throttle

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #24169:
URL: https://github.com/apache/beam/issues/24169#issuecomment-1535010025

   @chamikaramj Is your PR supposed to write the appropriate error messages?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] stfines-clgx commented on issue #24169: [Feature Request]: BigQuery Storage API read throttle

Posted by "stfines-clgx (via GitHub)" <gi...@apache.org>.
stfines-clgx commented on issue #24169:
URL: https://github.com/apache/beam/issues/24169#issuecomment-1522434454

   as the actual GCP customer with the issue - it continues to occur in 2.46.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] kennknowles closed issue #24169: [Feature Request]: BigQuery Storage API read throttle

Posted by GitBox <gi...@apache.org>.
kennknowles closed issue #24169: [Feature Request]: BigQuery Storage API read throttle
URL: https://github.com/apache/beam/issues/24169


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org