You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2023/01/20 03:49:33 UTC

[GitHub] [beam] prodriguezdefino opened a new pull request, #25094: Externalizing the StreamWriter parameters for StorageWrites

prodriguezdefino opened a new pull request, #25094:
URL: https://github.com/apache/beam/pull/25094

   This change adds the possibility of configuring the amount of inflight messages or bytes the underlying StreamWriter holds at the connection level before start channeling the new request to a different connection, when using connection pool for storage writes. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] reuvenlax merged pull request #25094: Externalizing the StreamWriter parameters for StorageWrites

Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax merged PR #25094:
URL: https://github.com/apache/beam/pull/25094


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] reuvenlax commented on a diff in pull request #25094: Externalizing the StreamWriter parameters for StorageWrites

Posted by GitBox <gi...@apache.org>.
reuvenlax commented on code in PR #25094:
URL: https://github.com/apache/beam/pull/25094#discussion_r1082144502


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryOptions.java:
##########
@@ -109,6 +109,20 @@
 
   void setNumStorageWriteApiStreamAppendClients(Integer value);
 
+  @Description(
+      "The max number of messages inflight that we expect each connection will have when connection pool usage is enabled for storage writes.")
+  @Default.Long(1000)
+  Long getStorageWriteMaxInflightRequests();
+
+  void setStorageWriteMaxInflightRequests(Long value);
+
+  @Description(
+      "The max size in bytes for inflight messages that we expect each connection will have when connection pool usage is enabled for storage writes.")

Review Comment:
   Nit: this takes effect whether connection pooling is enabled or not.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] prodriguezdefino commented on a diff in pull request #25094: Externalizing the StreamWriter parameters for StorageWrites

Posted by GitBox <gi...@apache.org>.
prodriguezdefino commented on code in PR #25094:
URL: https://github.com/apache/beam/pull/25094#discussion_r1082184017


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryOptions.java:
##########
@@ -109,6 +109,20 @@
 
   void setNumStorageWriteApiStreamAppendClients(Integer value);
 
+  @Description(
+      "The max number of messages inflight that we expect each connection will have when connection pool usage is enabled for storage writes.")
+  @Default.Long(1000)
+  Long getStorageWriteMaxInflightRequests();
+
+  void setStorageWriteMaxInflightRequests(Long value);
+
+  @Description(
+      "The max size in bytes for inflight messages that we expect each connection will have when connection pool usage is enabled for storage writes.")

Review Comment:
   corrected.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] prodriguezdefino commented on pull request #25094: Externalizing the StreamWriter parameters for StorageWrites

Posted by GitBox <gi...@apache.org>.
prodriguezdefino commented on PR #25094:
URL: https://github.com/apache/beam/pull/25094#issuecomment-1397893401

   R: @reuvenlax 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] reuvenlax commented on pull request #25094: Externalizing the StreamWriter parameters for StorageWrites

Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on PR #25094:
URL: https://github.com/apache/beam/pull/25094#issuecomment-1411380465

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] github-actions[bot] commented on pull request #25094: Externalizing the StreamWriter parameters for StorageWrites

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #25094:
URL: https://github.com/apache/beam/pull/25094#issuecomment-1397894081

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] reuvenlax commented on pull request #25094: Externalizing the StreamWriter parameters for StorageWrites

Posted by "reuvenlax (via GitHub)" <gi...@apache.org>.
reuvenlax commented on PR #25094:
URL: https://github.com/apache/beam/pull/25094#issuecomment-1399628448

   Run Java_GCP_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] prodriguezdefino commented on pull request #25094: Externalizing the StreamWriter parameters for StorageWrites

Posted by "prodriguezdefino (via GitHub)" <gi...@apache.org>.
prodriguezdefino commented on PR #25094:
URL: https://github.com/apache/beam/pull/25094#issuecomment-1411322151

   Run Java_GCP_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org