You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 22:50:45 UTC
[GitHub] [beam] damccorm opened a new issue, #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
damccorm opened a new issue, #21368:
URL: https://github.com/apache/beam/issues/21368
When looking at thread stacks for a pipeline reading pubsub and writing to BQ using the WriteApi there were 3600 Gax- prefixed threads.
The # of threads an possibly performance could be improved by finding where these came from and using a shared executor.
Imported from Jira [BEAM-13826](https://issues.apache.org/jira/browse/BEAM-13826). Original Jira may contain additional context.
Reported by: scwhittle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1227473281
That would make sense. Would need to check that the objects being managed there aren't stateful in a meaningful way that would impact being used across multiple threads (e.g. have state transitions like open->closed, buffer data for a bundle, ...) allowing the object to be marked `@ThreadSafe` making it bundle agnostic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1227443379
@lukecwik do you have a recommendation on executors to try?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1290967641
@reuvenlax it looks like you then went and rolled forward the fix? https://github.com/apache/beam/pull/23795
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1344582938
agreed. The underlying gax library has been updated, so when that is released we can re-implement the fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1225002206
IIUC the appropriate fix is to adjust client libraries used in various IOs to use a shared channel (@lukecwik can advise).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik closed issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
lukecwik closed issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
URL: https://github.com/apache/beam/issues/21368
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1251092829
https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/23996/
org.apache.beam.runners.fnexecution.control.RemoteExecutionTest mostly.
when I separated to just the scheduled executor, that was causing issues in BQ itself as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1228685241
Yes, but at least today there is no way of doing that AFAICT. We might have
to wait for the BigQuery support to land.
On Thu, Aug 25, 2022 at 1:20 PM johnjcasey ***@***.***> wrote:
> got it
> so we shouldn't add additional caching, but instead cache the transport
> s.t. all the various clients use the same one?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/issues/21368#issuecomment-1227719837>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFAYJVINEGMK7RTQ32VXRRTV27IPJANCNFSM5X4DNNQQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1274402238
Right now if they are writing to many tables using dynamic destinations, we
must have a separate grpc channel for each table stream. The BigQuery team
is working on multiplexing support which would allow using a single grpc
for all tables, however that is not quite ready yet.
On Wed, Aug 24, 2022 at 2:00 PM Brian Hulette ***@***.***>
wrote:
> Is there any documentation on where we use GAX, or a pointer to a design
> pattern for this?
>
> I'm not sure about documentation. My understanding from offline discussion
> with @lukecwik <https://github.com/lukecwik>, is that gax-java is used by
> grpc-java which is used by Google client libraries used by various GCP IOs.
> The client libraries generally have ways to specify an executor that would
> allow us to specify a shared one. See here
> <https://github.com/googleapis/gax-java/issues/1286#issuecomment-829555309>
> for an example in the Spanner client.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/issues/21368#issuecomment-1226333772>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFAYJVNDVLF754HHGSFFVUTV22ENDANCNFSM5X4DNNQQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1292401020
Correct - the PR only broke BigQueryIO.
On Wed, Oct 26, 2022 at 10:49 AM Chamikara Jayalath <
***@***.***> wrote:
> Thanks. Merged the cherry-pick.
>
> Removing this from the 2.43.0 milestone.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/issues/21368#issuecomment-1292396790>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFAYJVPC57RRROZBRVUA4PDWFFVL3ANCNFSM5X4DNNQQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1290754176
Hi folks, any updates here ?
This is currently blocking the 2.43.0 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1227696403
We do cache the client instances. The problem is that today, each
destination table requires a separate client.
On Thu, Aug 25, 2022 at 8:45 AM Lukasz Cwik ***@***.***>
wrote:
> Yes the scheduled executor service should be shared but it is the wrong
> scope of object to be shared. Based upon GCP client library best practices
> the recommendation is to share the client instance itself across multiple
> usages
> <https://cloud.google.com/apis/docs/client-libraries-best-practices#reuse_client_objects_and_sessions>
> .
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/issues/21368#issuecomment-1227451738>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFAYJVO3W6RHULH7JCN4ZGLV26IKRANCNFSM5X4DNNQQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1228740344
The client libraries allow injecting our own transport/threadpool via `BigQueryWriteClient#setTransportChannelProvider` and `BigQueryWriteClient#setBackgroundExecutorProvider`.
I still think we should be re-using the client across bundles by having a global object pool of `BQ Options -> Client`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1290933846
Was it reverted in the release branch as well ? (couldn't find a reference to a revert PR here).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1290937049
here is reuven's revert pr https://github.com/apache/beam/pull/23793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] scwhittle commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
scwhittle commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1234656194
Did you see Luke's suggestion about that?
```
The client libraries allow injecting our own transport/threadpool via BigQueryWriteClient#setTransportChannelProvider and BigQueryWriteClient#setBackgroundExecutorProvider.
```
So you could have a static shared transport and threadpool across all the cached clients
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1341294998
Any update on this? Given that it appears to be not data loss or total loss of functionality I am downgrading to P2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1227586896
Also, is there a way to close the client in a way where it cleans up those resources?
This would be useful to not leak instances of the client if the options change over time (e.g. dynamic destinations).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1227719837
got it
so we shouldn't add additional caching, but instead cache the transport s.t. all the various clients use the same one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1225746326
Is there any documentation on where we use GAX, or a pointer to a design pattern for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik closed issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
lukecwik closed issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
URL: https://github.com/apache/beam/issues/21368
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1245739873
@lukecwik or @scwhittle do either of you have a suggestion on what transport to use? I'm fairly unfamiliar with the details of these
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1249591679
What tests are they? I'm curious if there is some trivial pattern to them that is causing the test to fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1246954363
That makes sense, though we will have to do some testing to verify that changing from a 'normal' thread pool executor to a scheduled thread pool executor doesn't impact the rest of our GCP IOs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1290917551
This has been reverted, so the fix isn't in master at the moment, but also the overall Gax issue isn't resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1290937387
looks like it probably didn't make the release cut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] amiyalenka commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by "amiyalenka (via GitHub)" <gi...@apache.org>.
amiyalenka commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1418980371
Is this GAX Thread issue been resolved? If yes, which version having this fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1256381741
You can wrap the ScheduledExecutorService with one that wraps callables/runnables with one that sets countdown latch and outside of it waits for the latch to be set. This will ensure that the thread has started before the method returns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1253164508
The ThreadPoolExecutor implementation uses a synchronous queue meaning that scheduling a task will block till a thread takes it. Could this be the difference?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1227311884
Got it. I suppose I will need to do some research on executor options to know which one is the best for this use case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1226312180
@reuvenlax maybe you're thinking of https://github.com/apache/beam/pull/16619 which fixed a different thread explosion in `RetryManager`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1226292482
@reuvenlax do you have a link to the fix ?
also ccing @scwhittle who filed the original issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1258916954
Swapping libraries seems like it will be a much larger effort. Consider fixing up the much smaller scheduled executor service change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1274235758
This is not a library swap - it is part of the next version of the library
we already use. (already released, but not yet part of a BOM release).
On Mon, Sep 26, 2022 at 8:21 PM Lukasz Cwik ***@***.***>
wrote:
> Swapping libraries seems like it will be a much larger effort. Consider
> fixing up the much smaller scheduled executor service change.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/issues/21368#issuecomment-1258916954>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFAYJVLLLEQBKUTPOTNHZ53WAJR5NANCNFSM5X4DNNQQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1290956120
attempting it now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by "lukecwik (via GitHub)" <gi...@apache.org>.
lukecwik commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1419586569
2.45.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1256317231
That seems likely. I'm not sure how to configure a scheduled executor to replicate this pattern, or to change our system to handle the async scheduling
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1289375895
https://github.com/apache/beam/pull/23234#issuecomment-1287591156 says that this fix causes BQIO to hang during shutdown.
Re-opened this bug to triage and track impact to 2.43 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1227451738
Yes the scheduled executor service should be shared but it is the wrong scope of object to be shared. Based upon GCP client library best practices the recommendation is to [share the client instance itself across multiple usages](https://cloud.google.com/apis/docs/client-libraries-best-practices#reuse_client_objects_and_sessions).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1227466975
A somewhat simple change would be to update the various get methods in BigQueryServicesImpl to be cached. If you pass in the same options, you get the existing client instance.
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java#L183
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1229132117
We are reusing clients across bundles for streaming write API. The DoFns
that use this keep a global static cache of table name -> client
On Fri, Aug 26, 2022 at 10:19 AM Lukasz Cwik ***@***.***>
wrote:
> The client libraries allow injecting our own transport/threadpool via
> BigQueryWriteClient#setTransportChannelProvider and
> BigQueryWriteClient#setBackgroundExecutorProvider.
>
> I still think we should be re-using the client across bundles by having a
> global object pool of BQ Options -> Client
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/issues/21368#issuecomment-1228740344>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFAYJVPAHANGEPR2DAVA7LTV3D4DLANCNFSM5X4DNNQQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1224992732
John, assigning to you to prioritize since this could be hit by users when using various I/O connectors (for example, BigQueryIO sink with the STORAGE_WRITE_API mode).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1248731352
How do they assume this?
If its easy to clean-up I would prefer that we use only one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1261308798
So, once we solve for the schedule executor service issue, I don't think we can update to just providing an SES, because it would break the existing API that allows users to specify any executorservice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1258374030
Oh, that would be a good direction to investigate. Do you have a link to that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1225004653
I thought that the main bug had already been fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1226333772
> Is there any documentation on where we use GAX, or a pointer to a design pattern for this?
I'm not sure about documentation. My understanding from offline discussion with @lukecwik, is that gax-java is used by grpc-java which is used by Google client libraries used by various GCP IOs. The client libraries generally have ways to specify an executor that would allow us to specify a shared one. See [here](https://github.com/googleapis/gax-java/issues/1286#issuecomment-829555309) for an example in the Spanner client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1230485311
I believe that to be sufficient caching from a client level.
Is there no way to have distinct clients share a transport, or a transport provider?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
lukecwik commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1246022651
There is only one gRPC channel provider, the InstantiatingGrpcChannelProvider which is also the default. You could experiment with creating a fixed one but I would suggest using `setBackgroundExecutorProvider` on all the clients using a FixedExecutorService and updating https://github.com/apache/beam/blob/9be9a43c1b85be9b2f78e2943f6092ccb88e13b4/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.java#L136 to return a ScheduledExecutorService which is supplied to the FixedExecutorService.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1227728171
Speaking of, where does that caching occur?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1258367452
Note that BigQuery now has a version of their library that allows sharing
multiple table writes over the same connection.
On Fri, Sep 23, 2022 at 8:49 AM Lukasz Cwik ***@***.***>
wrote:
> You can wrap the ScheduledExecutorService with one that wraps
> callables/runnables with one that sets countdown latch and outside of it
> waits for the latch to be set. This will ensure that the thread has started
> before the method returns.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/issues/21368#issuecomment-1256381741>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFAYJVPR4L6Y26GFH57WBLTV7XGRFANCNFSM5X4DNNQQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1248385498
It looks like a variety of our tests assume the ThreadPoolExecutor instead of a scheduled one, and time out when a scheduled executor is default. I believe the solution will be to instead provide a second thread pool for places that need scheduled execution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1249370238
It isn't explicit, but we have 14 tests that fail with varieties of timeouts when we use the scheduled executor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1290939924
Ack. Please send a cherry-pick. (or I can start one).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1292239382
looks like the roll forward was partial. @chamikaramj here is the cherry pick https://github.com/apache/beam/pull/23854
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1290970427
@reuvenlax can you clarify if master is good, or what the current state is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on issue #21368: Reduce number of Gax related threads, likely by providing common executor to GAX clients
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #21368:
URL: https://github.com/apache/beam/issues/21368#issuecomment-1292396790
Thanks. Merged the cherry-pick.
Removing this from the 2.43.0 milestone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org