You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/07/22 19:20:46 UTC

[GitHub] [beam] prodriguezdefino opened a new issue, #22416: [Bug]: Always propagate successful results in BQ StreamingInserts seems to reduce performance

prodriguezdefino opened a new issue, #22416:
URL: https://github.com/apache/beam/issues/22416

   ### What happened?
   
   After the inclusion of the propagation of successful inserts in v2.37.0 pipelines that were able to handle very large throughputs (+10GB/s) started to see problems when recovering from backlog or when delays happened from upstream and there were already at max num of workers. 
   
   It seems that holding the TupleTag data after insertion generates some pressure on the workers that slowdown the pipeline reducing the max throughput at a given number of user workers. 
   
   Introducing an option to enable/disable this propagation (enabled by default for backwards compatibility) should help to recover part of that lost throughput on large volume pipelines. 
   
   ### Issue Priority
   
   Priority: 1
   
   ### Issue Component
   
   Component: io-java-gcp


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] prodriguezdefino closed issue #22416: [Bug]: Always propagate successful results in BQ StreamingInserts seems to reduce performance

Posted by GitBox <gi...@apache.org>.
prodriguezdefino closed issue #22416: [Bug]: Always propagate successful results in BQ StreamingInserts seems to reduce performance
URL: https://github.com/apache/beam/issues/22416


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] prodriguezdefino commented on issue #22416: [Bug]: Always propagate successful results in BQ StreamingInserts seems to reduce performance

Posted by GitBox <gi...@apache.org>.
prodriguezdefino commented on issue #22416:
URL: https://github.com/apache/beam/issues/22416#issuecomment-1192872448

   .add-labels java,gcp,bigquery


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] prodriguezdefino commented on issue #22416: [Bug]: Always propagate successful results in BQ StreamingInserts seems to reduce performance

Posted by GitBox <gi...@apache.org>.
prodriguezdefino commented on issue #22416:
URL: https://github.com/apache/beam/issues/22416#issuecomment-1192870011

   .take-issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] prodriguezdefino commented on issue #22416: [Bug]: Always propagate successful results in BQ StreamingInserts seems to reduce performance

Posted by GitBox <gi...@apache.org>.
prodriguezdefino commented on issue #22416:
URL: https://github.com/apache/beam/issues/22416#issuecomment-1192871984

   .remove-labels 'awaiting triage'
   .add-labels java,gcp,bigquery


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org