You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2023/01/20 03:49:33 UTC
[GitHub] [beam] prodriguezdefino opened a new pull request, #25094: Externalizing the StreamWriter parameters for StorageWrites
prodriguezdefino opened a new pull request, #25094:
URL: https://github.com/apache/beam/pull/25094
This change adds the possibility of configuring the amount of inflight messages or bytes the underlying StreamWriter holds at the connection level before start channeling the new request to a different connection, when using connection pool for storage writes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax merged pull request #25094: Externalizing the StreamWriter parameters for StorageWrites
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax merged PR #25094:
URL: https://github.com/apache/beam/pull/25094
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on a diff in pull request #25094: Externalizing the StreamWriter parameters for StorageWrites
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on code in PR #25094:
URL: https://github.com/apache/beam/pull/25094#discussion_r1082144502
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryOptions.java:
##########
@@ -109,6 +109,20 @@
void setNumStorageWriteApiStreamAppendClients(Integer value);
+ @Description(
+ "The max number of messages inflight that we expect each connection will have when connection pool usage is enabled for storage writes.")
+ @Default.Long(1000)
+ Long getStorageWriteMaxInflightRequests();
+
+ void setStorageWriteMaxInflightRequests(Long value);
+
+ @Description(
+ "The max size in bytes for inflight messages that we expect each connection will have when connection pool usage is enabled for storage writes.")
Review Comment:
Nit: this takes effect whether connection pooling is enabled or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] prodriguezdefino commented on a diff in pull request #25094: Externalizing the StreamWriter parameters for StorageWrites
Posted by GitBox <gi...@apache.org>.
prodriguezdefino commented on code in PR #25094:
URL: https://github.com/apache/beam/pull/25094#discussion_r1082184017
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryOptions.java:
##########
@@ -109,6 +109,20 @@
void setNumStorageWriteApiStreamAppendClients(Integer value);
+ @Description(
+ "The max number of messages inflight that we expect each connection will have when connection pool usage is enabled for storage writes.")
+ @Default.Long(1000)
+ Long getStorageWriteMaxInflightRequests();
+
+ void setStorageWriteMaxInflightRequests(Long value);
+
+ @Description(
+ "The max size in bytes for inflight messages that we expect each connection will have when connection pool usage is enabled for storage writes.")
Review Comment:
corrected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] prodriguezdefino commented on pull request #25094: Externalizing the StreamWriter parameters for StorageWrites
Posted by GitBox <gi...@apache.org>.
prodriguezdefino commented on PR #25094:
URL: https://github.com/apache/beam/pull/25094#issuecomment-1397893401
R: @reuvenlax
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #25094: Externalizing the StreamWriter parameters for StorageWrites
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on PR #25094:
URL: https://github.com/apache/beam/pull/25094#issuecomment-1411380465
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #25094: Externalizing the StreamWriter parameters for StorageWrites
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #25094:
URL: https://github.com/apache/beam/pull/25094#issuecomment-1397894081
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #25094: Externalizing the StreamWriter parameters for StorageWrites
Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on PR #25094:
URL: https://github.com/apache/beam/pull/25094#issuecomment-1399628448
Run Java_GCP_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] prodriguezdefino commented on pull request #25094: Externalizing the StreamWriter parameters for StorageWrites
Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #25094:
URL: https://github.com/apache/beam/pull/25094#issuecomment-1411322151
Run Java_GCP_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org