You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "chenboat (via GitHub)" <gi...@apache.org> on 2023/03/29 20:33:59 UTC
[GitHub] [pinot] chenboat opened a new pull request, #10507: Allow configurable socket timeout during segment downloads.
chenboat opened a new pull request, #10507:
URL: https://github.com/apache/pinot/pull/10507
Today the socket timeout during segment download is hardcoded to be 10mins. In some large use cases this timeout is too long and result in connection pool exhaustion. This PR makes the timeout configurable.
Instructions:
1. The PR has to be tagged with at least one of the following labels (*):
1. `feature`
2. `bugfix`
3. `performance`
4. `ui`
5. `backward-incompat`
6. `release-notes` (**)
2. Remove these instructions before publishing the PR.
(*) Other labels to consider:
- `testing`
- `dependencies`
- `docker`
- `kubernetes`
- `observability`
- `security`
- `code-style`
- `extension-point`
- `refactor`
- `cleanup`
(**) Use `release-notes` label for scenarios like:
- New configuration options
- Deprecation of configurations
- Signature changes to public methods/interfaces
- New plugins added or old plugins removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jackjlli commented on a diff in pull request #10507: Allow configurable socket timeout during segment downloads.
Posted by "jackjlli (via GitHub)" <gi...@apache.org>.
jackjlli commented on code in PR #10507:
URL: https://github.com/apache/pinot/pull/10507#discussion_r1152493597
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java:
##########
@@ -1130,6 +1130,22 @@ public int downloadFile(URI uri, File dest)
return downloadFile(uri, dest, null);
}
+ /**
+ * Download a file with a configurable timeout
+ *
+ * @param uri URI
+ * @param dest File destination
+ * @param authProvider auth provider
+ * @param socketTimeoutMs connection socket timeout ms
+ * @return Response status code
+ * @throws IOException
+ * @throws HttpErrorStatusException
+ */
+ public int downloadFile(URI uri, File dest, AuthProvider authProvider, int socketTimeoutMs)
Review Comment:
nit: could you change the Line 1161 to invoke this method in order to avoid duplicate logic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10507: Allow configurable socket timeout during segment downloads.
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #10507:
URL: https://github.com/apache/pinot/pull/10507#discussion_r1153629910
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/FileUploadDownloadClient.java:
##########
@@ -1149,16 +1165,17 @@ public int downloadFile(URI uri, File dest, AuthProvider authProvider)
* Download a file.
*
* @param uri URI
+ * @param socketTimeoutMs socketTimeoutMs
* @param dest File destination
* @param authProvider auth provider
* @param httpHeaders http headers
* @return Response status code
* @throws IOException
* @throws HttpErrorStatusException
*/
- public int downloadFile(URI uri, File dest, AuthProvider authProvider, List<Header> httpHeaders)
+ public int downloadFile(URI uri, int socketTimeoutMs, File dest, AuthProvider authProvider, List<Header> httpHeaders)
Review Comment:
Let's keep this signature as is, and add a new one which takes `socketTimeoutMs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org