You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "reuvenlax (via GitHub)" <gi...@apache.org> on 2026/04/29 02:12:18 UTC
[PR] Enable connection pooling by default [beam]
reuvenlax opened a new pull request, #38313:
URL: https://github.com/apache/beam/pull/38313
This was initially disabled as there were stability issues with connection pools. Several years have now gone by, and connection pooling support is stable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-5692660354
I'm talking to the BigQuery team, and we have some theories as to what might be happening. Will circle back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-4340384498
R: @ahmedabu98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-5606002162
ping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-4499800048
Is this ready to be merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-5914648323
is this stalled or hopeless or... ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-4915994854
Hi @reuvenlax, any further movement on this? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-4606457890
There is a remaining failure that I'm discussing with the BigQuery Storage
API team.
On Mon, Jun 1, 2026 at 4:11 PM Chamikara Jayalath ***@***.***>
wrote:
> *chamikaramj* left a comment (apache/beam#38313)
> <https://github.com/apache/beam/pull/38313#issuecomment-4597225672>
>
> Seems like this is ready to be merged @reuvenlax
> <https://github.com/reuvenlax> ?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/38313?email_source=notifications&email_token=AFAYJVNJTQ4LOXL34G4PRA345YETRA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINJZG4ZDENJWG4ZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4597225672>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFAYJVL4QPLHOFIFEKKQ6W345YETRAVCNFSM6AAAAACYKEN7ESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DKOJXGIZDKNRXGI>
> .
> Triage notifications, keep track of coding agent tasks and review pull
> requests on the go with GitHub Mobile for iOS
> <https://github.com/notifications/mobile/ios/AFAYJVMMT4LUTLZQAJ4WOVT45YETRA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINJZG4ZDENJWG4ZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y>
> and Android
> <https://github.com/notifications/mobile/android/AFAYJVPQD5DWUD2VK4TUGLL45YETRA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINJZG4ZDENJWG4ZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>.
> Download it today!
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-5680814698
I have a couple of questions:
1. Do you know why connection pooling makes this worse? That's not intuitively obvious to me
2. Is there a way to update the test such that the update eventually does happen? If the update truly never happens, then that seems problematic (even if the current behavior is relying on lucky outcomes on the BQ side)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-5697882288
> @damccorm As to your suggestion, I'm not sure if Beam has a good way of knowing whether an option was explicitly set. i.e. if we change the default, BigQueryOptions.getUseStorageApiConnectionPool() == true could not distinguish between a user who explicitly set the option and one who is getting the new default. If we always turn connection pooling off when AutoSchemaUpdate is used, we may regress performance for existing users who are explicitly enabling connection pooling.
Could we make the option Nullable (and default null)? That gives us 3 states:
- null (nothing set)
- true (set by user)
- false (set by user)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-5685707988
@damccorm
1. Good question, I'm still chatting with BQ about this. I will say that the current test waits until the table.get() returns the new schema, however that doesn't guarantee that the write-api servers have seen the update. I do see that we generally see updates from write-api about _some_ of the tables, but generally not all. Connection pooling means that updates for many tables are multiplexed on a single RPC, and the schema update comes from the update response. It's possible that this changes timing enough to break the test. It's also possible that there's a bug in BigQuery around schema updates and connection pooling.
2. I'm not sure how to make this test fundamentally not flaky. We can query write api for the schema and wait until we see that updated. However with the default stream, the query might go to any of.a pool of write-api frontend servers, which means that the test might still be flaky. I do have a pending PR that adds a "high consistency" mode to schema updates - i.e. it guarantees that we don't ever miss updates. However this mode is more expensive, and unlikely to be the default mode for many users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-5687960065
I wonder if we should just leave connection pooling off for AutoSchemaUpdate cases (unless the user explicitly sets it)?
The downside is additional complexity in our implementation, but until we're sure that this doesn't break or significantly degrade the feature, I'm a little concerned about turning it on by default. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-5692697152
@damccorm As to your suggestion, I'm not sure if Beam has a good way of knowing whether an option was explicitly set. i.e. if we change the default, BigQueryOptions.getUseStorageApiConnectionPool() == true could not distinguish between a user who explicitly set the option and one who is getting the new default. If we always turn connection pooling off when AutoSchemaUpdate is used, we may regress performance for existing users who are explicitly enabling connection pooling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-4340368826
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-4340365804
R: @ahmedabu98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-4597225672
Seems like this is ready to be merged @reuvenlax ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-5648373881
@chamikaramj @damccorm @ahmedabu98 this causes StorageApiSinkSchemaUpdateIT.testAtLeastOnceDynamicDestinationsWithAutoSchemaUpdate to fail.
AFAICT from debugging, the tests fail because we don't get a schema-update response back from BigQuery before the test finishes. I'll follow up with BQ team to ensure there is no bug here, but truth is that this response was never guaranteed to be synchronous (there's caching in the write API, etc.); this test was already potentially flaky, and I suspect that this change just pushed it into failing more consistently.
My current inclination is to just force connection pooling off for this test, and merge the PR. What do you all think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-4340365511
## Summary of Changes
Hello, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request enables connection pooling by default for the BigQuery Storage API. Given that the underlying stability issues that previously necessitated disabling this feature have been resolved over time, it is now considered safe and recommended to enable this by default to improve performance.
### Highlights
* **Default Configuration Change**: Updated the default value for BigQuery Storage API connection pooling from false to true.
🧠 **New Feature in Public Preview:** You can now enable **Memory** to help **Gemini Code Assist** learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. **Click [here](https://codeassist.google/code-review/login) to enable Memory in your admin console.**
<details>
<summary><b>Using Gemini Code Assist</b></summary>
<br>
The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.
<b>Invoking Gemini</b>
You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page.
Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help | `/gemini help` | Displays a list of available commands.
<b>Customization</b>
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).
<b>Limitations & Feedback</b>
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG).
</details>
[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-4500721904
> Is this ready to be merged?
Rerunning tests. they have been failing before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-4549991664
> java.lang.IllegalArgumentException: useStorageApiConnectionPool only supported when using STORAGE_API_AT_LEAST_ONCE at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
>
> maybe default should be factory that will look at method name and set it to true only if method is STORAGE_API_AT_LEAST_ONCE
Good catch - fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-4543966325
java.lang.IllegalArgumentException: useStorageApiConnectionPool only supported when using STORAGE_API_AT_LEAST_ONCE
at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
maybe default should be factory that will look at method name and set it to true only if method is STORAGE_API_AT_LEAST_ONCE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-5570845463
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable connection pooling by default [beam]
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on PR #38313:
URL: https://github.com/apache/beam/pull/38313#issuecomment-5914709029
It does look like we're hitting a bug in BigQuery. The bug only affects a
somewhat uncommon feature in Beam, but I've been working through it with
the BQ team.
On Wed, Sep 30, 2026 at 8:44 AM Kenn Knowles ***@***.***>
wrote:
> *kennknowles* left a comment (apache/beam#38313)
> <https://github.com/apache/beam/pull/38313#issuecomment-5914648323>
>
> is this stalled or hopeless or... ?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/38313?email_source=notifications&email_token=AFAYJVN7EXNLVXUEZYDMKID5RUS5HA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKOJRGQ3DIOBTGIZ2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5914648323>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFAYJVMY5SO5YG26AX5IABL5RUS5HAVCNFSNUABEKJSXA33TNF2G64TZHM2TAOJQGQZDINJ3JFZXG5LFHM2DGNBXGQYDMNRZGSQXMAQ>
> .
> Triage notifications, keep track of coding agent tasks and review pull
> requests on the go with GitHub Mobile for iOS
> <https://github.com/notifications/mobile/ios/AFAYJVKRDHOCUBAU6ASLFJT5RUS5HA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKOJRGQ3DIOBTGIZ2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y>
> and Android
> <https://github.com/notifications/mobile/android/AFAYJVKHHG5MOHFYVVIETID5RUS5HA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKOJRGQ3DIOBTGIZ2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>.
> Download it today!
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org