You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/03/09 19:46:48 UTC
[GitHub] [pinot] walterddr opened a new pull request #8329: using HttpClient
walterddr opened a new pull request #8329:
URL: https://github.com/apache/pinot/pull/8329
alternative solution to #8293
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #8329: using HttpClient
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#issuecomment-1063683490
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8329](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (035e67f) into [master](https://codecov.io/gh/apache/pinot/commit/8ef4dbacfb42596590d08c7c6f14f88093dfe7d5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8ef4dba) will **decrease** coverage by `9.71%`.
> The diff coverage is `41.47%`.
> :exclamation: Current head 035e67f differs from pull request most recent head 001e1b9. Consider uploading reports for the commit 001e1b9 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #8329 +/- ##
============================================
- Coverage 70.80% 61.08% -9.72%
+ Complexity 4258 4174 -84
============================================
Files 1636 1626 -10
Lines 85804 85532 -272
Branches 12920 12887 -33
============================================
- Hits 60755 52250 -8505
- Misses 20858 29444 +8586
+ Partials 4191 3838 -353
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `27.57% <36.86%> (+0.05%)` | :arrow_up: |
| unittests1 | `66.87% <23.29%> (-0.08%)` | :arrow_down: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...inot/controller/helix/ControllerRequestClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9Db250cm9sbGVyUmVxdWVzdENsaWVudC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ion/tasks/BaseSingleSegmentConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVNpbmdsZVNlZ21lbnRDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-74.61%)` | :arrow_down: |
| [...ot/plugin/minion/tasks/SegmentConversionUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvU2VnbWVudENvbnZlcnNpb25VdGlscy5qYXZh) | `38.33% <ø> (-38.34%)` | :arrow_down: |
| [...he/pinot/segment/local/utils/SegmentPushUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9TZWdtZW50UHVzaFV0aWxzLmphdmE=) | `13.48% <0.00%> (ø)` | |
| [...org/apache/pinot/common/utils/http/HttpClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaHR0cC9IdHRwQ2xpZW50LmphdmE=) | `45.45% <45.45%> (ø)` | |
| [...e/pinot/common/utils/FileUploadDownloadClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRmlsZVVwbG9hZERvd25sb2FkQ2xpZW50LmphdmE=) | `56.93% <60.00%> (-6.08%)` | :arrow_down: |
| [...n/java/org/apache/pinot/common/utils/TlsUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvVGxzVXRpbHMuamF2YQ==) | `76.40% <60.00%> (-1.25%)` | :arrow_down: |
| [...ger/realtime/Server2ControllerSegmentUploader.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VydmVyMkNvbnRyb2xsZXJTZWdtZW50VXBsb2FkZXIuamF2YQ==) | `72.41% <100.00%> (ø)` | |
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | `48.57% <100.00%> (-2.86%)` | :arrow_down: |
| [.../tasks/BaseMultipleSegmentsConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZU11bHRpcGxlU2VnbWVudHNDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `69.16% <100.00%> (-20.84%)` | :arrow_down: |
| ... and [299 more](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8ef4dba...001e1b9](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] yupeng9 commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r828188238
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,371 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendGetRequest(URI uri)
Review comment:
why do we add deprecated method, given this is a new file
##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/ControllerRequestClient.java
##########
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.tenant.Tenant;
+import org.apache.pinot.spi.config.tenant.TenantRole;
+import org.apache.pinot.spi.data.Schema;
+
+
+public class ControllerRequestClient {
Review comment:
need docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#issuecomment-1063683490
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8329](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (58eaef1) into [master](https://codecov.io/gh/apache/pinot/commit/98d781092136eeb3a80e9e54fc8736f4c3d270f5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (98d7810) will **increase** coverage by `2.85%`.
> The diff coverage is `17.88%`.
```diff
@@ Coverage Diff @@
## master #8329 +/- ##
============================================
+ Coverage 64.08% 66.94% +2.85%
+ Complexity 4254 4186 -68
============================================
Files 1592 1238 -354
Lines 83916 62624 -21292
Branches 12706 9739 -2967
============================================
- Hits 53779 41923 -11856
+ Misses 26261 17718 -8543
+ Partials 3876 2983 -893
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests1 | `66.94% <17.88%> (-0.02%)` | :arrow_down: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | `6.66% <0.00%> (ø)` | |
| [...he/pinot/segment/local/utils/SegmentPushUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9TZWdtZW50UHVzaFV0aWxzLmphdmE=) | `13.48% <0.00%> (ø)` | |
| [...org/apache/pinot/common/utils/http/HttpClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaHR0cC9IdHRwQ2xpZW50LmphdmE=) | `14.46% <14.46%> (ø)` | |
| [...e/pinot/common/utils/FileUploadDownloadClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRmlsZVVwbG9hZERvd25sb2FkQ2xpZW50LmphdmE=) | `18.07% <22.85%> (+1.96%)` | :arrow_up: |
| [...n/java/org/apache/pinot/common/utils/TlsUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvVGxzVXRpbHMuamF2YQ==) | `4.21% <36.36%> (+4.21%)` | :arrow_up: |
| [...ger/realtime/Server2ControllerSegmentUploader.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VydmVyMkNvbnRyb2xsZXJTZWdtZW50VXBsb2FkZXIuamF2YQ==) | `72.41% <100.00%> (ø)` | |
| [.../aggregation/function/ModeAggregationFunction.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9Nb2RlQWdncmVnYXRpb25GdW5jdGlvbi5qYXZh) | `87.56% <0.00%> (-1.09%)` | :arrow_down: |
| [...ot/segment/spi/creator/SegmentGeneratorConfig.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvU2VnbWVudEdlbmVyYXRvckNvbmZpZy5qYXZh) | `82.08% <0.00%> (-0.98%)` | :arrow_down: |
| [...he/pinot/segment/local/utils/TableConfigUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9UYWJsZUNvbmZpZ1V0aWxzLmphdmE=) | `65.34% <0.00%> (-0.80%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerGauge.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyR2F1Z2UuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| ... and [363 more](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [98d7810...58eaef1](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #8329: using HttpClient
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#issuecomment-1063683490
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8329](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e288a0b) into [master](https://codecov.io/gh/apache/pinot/commit/8ef4dbacfb42596590d08c7c6f14f88093dfe7d5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8ef4dba) will **decrease** coverage by `1.26%`.
> The diff coverage is `41.70%`.
[](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #8329 +/- ##
============================================
- Coverage 70.80% 69.53% -1.27%
+ Complexity 4258 4255 -3
============================================
Files 1636 1638 +2
Lines 85804 85899 +95
Branches 12920 12927 +7
============================================
- Hits 60755 59734 -1021
- Misses 20858 22000 +1142
+ Partials 4191 4165 -26
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `27.65% <34.78%> (+0.14%)` | :arrow_up: |
| unittests1 | `66.85% <22.04%> (-0.10%)` | :arrow_down: |
| unittests2 | `14.17% <3.40%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...not/ingestion/common/DefaultControllerRestApi.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vdjBfZGVwcmVjYXRlZC9waW5vdC1pbmdlc3Rpb24tY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9pbmdlc3Rpb24vY29tbW9uL0RlZmF1bHRDb250cm9sbGVyUmVzdEFwaS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ion/tasks/BaseSingleSegmentConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVNpbmdsZVNlZ21lbnRDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `3.17% <0.00%> (-71.43%)` | :arrow_down: |
| [...ot/plugin/minion/tasks/SegmentConversionUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvU2VnbWVudENvbnZlcnNpb25VdGlscy5qYXZh) | `38.33% <ø> (-38.34%)` | :arrow_down: |
| [...he/pinot/segment/local/utils/SegmentPushUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9TZWdtZW50UHVzaFV0aWxzLmphdmE=) | `13.48% <0.00%> (ø)` | |
| [...inot/controller/helix/ControllerRequestClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9Db250cm9sbGVyUmVxdWVzdENsaWVudC5qYXZh) | `19.04% <19.04%> (ø)` | |
| [...org/apache/pinot/common/utils/http/HttpClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaHR0cC9IdHRwQ2xpZW50LmphdmE=) | `41.66% <41.66%> (ø)` | |
| [...e/pinot/common/utils/FileUploadDownloadClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRmlsZVVwbG9hZERvd25sb2FkQ2xpZW50LmphdmE=) | `56.93% <60.00%> (-6.08%)` | :arrow_down: |
| [...n/java/org/apache/pinot/common/utils/TlsUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvVGxzVXRpbHMuamF2YQ==) | `76.40% <60.00%> (-1.25%)` | :arrow_down: |
| [...ger/realtime/Server2ControllerSegmentUploader.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VydmVyMkNvbnRyb2xsZXJTZWdtZW50VXBsb2FkZXIuamF2YQ==) | `72.41% <100.00%> (ø)` | |
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | `48.57% <100.00%> (-2.86%)` | :arrow_down: |
| ... and [118 more](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8ef4dba...e288a0b](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #8329: using HttpClient
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#issuecomment-1063683490
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8329](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e288a0b) into [master](https://codecov.io/gh/apache/pinot/commit/8ef4dbacfb42596590d08c7c6f14f88093dfe7d5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8ef4dba) will **decrease** coverage by `56.62%`.
> The diff coverage is `3.40%`.
[](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #8329 +/- ##
=============================================
- Coverage 70.80% 14.17% -56.63%
+ Complexity 4258 81 -4177
=============================================
Files 1636 1593 -43
Lines 85804 84015 -1789
Branches 12920 12723 -197
=============================================
- Hits 60755 11912 -48843
- Misses 20858 71207 +50349
+ Partials 4191 896 -3295
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.17% <3.40%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...e/pinot/common/utils/FileUploadDownloadClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRmlsZVVwbG9hZERvd25sb2FkQ2xpZW50LmphdmE=) | `0.00% <0.00%> (-63.01%)` | :arrow_down: |
| [...n/java/org/apache/pinot/common/utils/TlsUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvVGxzVXRpbHMuamF2YQ==) | `0.00% <0.00%> (-77.65%)` | :arrow_down: |
| [...org/apache/pinot/common/utils/http/HttpClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaHR0cC9IdHRwQ2xpZW50LmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...ger/realtime/Server2ControllerSegmentUploader.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VydmVyMkNvbnRyb2xsZXJTZWdtZW50VXBsb2FkZXIuamF2YQ==) | `0.00% <0.00%> (-72.42%)` | :arrow_down: |
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-51.43%)` | :arrow_down: |
| [...not/ingestion/common/DefaultControllerRestApi.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vdjBfZGVwcmVjYXRlZC9waW5vdC1pbmdlc3Rpb24tY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9pbmdlc3Rpb24vY29tbW9uL0RlZmF1bHRDb250cm9sbGVyUmVzdEFwaS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../tasks/BaseMultipleSegmentsConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZU11bHRpcGxlU2VnbWVudHNDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `3.33% <0.00%> (-86.67%)` | :arrow_down: |
| [...ion/tasks/BaseSingleSegmentConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVNpbmdsZVNlZ21lbnRDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `3.17% <0.00%> (-71.43%)` | :arrow_down: |
| [...ot/plugin/minion/tasks/SegmentConversionUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvU2VnbWVudENvbnZlcnNpb25VdGlscy5qYXZh) | `0.00% <ø> (-76.67%)` | :arrow_down: |
| [...he/pinot/segment/local/utils/SegmentPushUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9TZWdtZW50UHVzaFV0aWxzLmphdmE=) | `0.00% <0.00%> (-13.49%)` | :arrow_down: |
| ... and [1313 more](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8ef4dba...e288a0b](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #8329: using HttpClient
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#issuecomment-1063683490
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8329](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6f2d43d) into [master](https://codecov.io/gh/apache/pinot/commit/8ef4dbacfb42596590d08c7c6f14f88093dfe7d5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8ef4dba) will **decrease** coverage by `6.78%`.
> The diff coverage is `26.79%`.
[](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #8329 +/- ##
============================================
- Coverage 70.80% 64.02% -6.79%
+ Complexity 4258 4253 -5
============================================
Files 1636 1592 -44
Lines 85804 83933 -1871
Branches 12920 12715 -205
============================================
- Hits 60755 53737 -7018
- Misses 20858 26324 +5466
+ Partials 4191 3872 -319
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `66.90% <28.08%> (-0.05%)` | :arrow_down: |
| unittests2 | `14.18% <0.00%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | `6.66% <0.00%> (-44.77%)` | :arrow_down: |
| [...not/ingestion/common/DefaultControllerRestApi.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vdjBfZGVwcmVjYXRlZC9waW5vdC1pbmdlc3Rpb24tY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9pbmdlc3Rpb24vY29tbW9uL0RlZmF1bHRDb250cm9sbGVyUmVzdEFwaS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../tasks/BaseMultipleSegmentsConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZU11bHRpcGxlU2VnbWVudHNDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `3.33% <0.00%> (-86.67%)` | :arrow_down: |
| [...ion/tasks/BaseSingleSegmentConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVNpbmdsZVNlZ21lbnRDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `3.17% <0.00%> (-71.43%)` | :arrow_down: |
| [...ot/plugin/minion/tasks/SegmentConversionUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvU2VnbWVudENvbnZlcnNpb25VdGlscy5qYXZh) | `0.00% <ø> (-76.67%)` | :arrow_down: |
| [...he/pinot/segment/local/utils/SegmentPushUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9TZWdtZW50UHVzaFV0aWxzLmphdmE=) | `13.48% <0.00%> (ø)` | |
| [...n/java/org/apache/pinot/common/utils/TlsUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvVGxzVXRpbHMuamF2YQ==) | `1.12% <20.00%> (-76.53%)` | :arrow_down: |
| [...e/pinot/common/utils/FileUploadDownloadClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRmlsZVVwbG9hZERvd25sb2FkQ2xpZW50LmphdmE=) | `14.85% <27.27%> (-48.16%)` | :arrow_down: |
| [...org/apache/pinot/common/utils/http/HttpClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaHR0cC9IdHRwQ2xpZW50LmphdmE=) | `30.00% <30.00%> (ø)` | |
| [...ger/realtime/Server2ControllerSegmentUploader.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VydmVyMkNvbnRyb2xsZXJTZWdtZW50VXBsb2FkZXIuamF2YQ==) | `72.41% <100.00%> (ø)` | |
| ... and [386 more](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8ef4dba...6f2d43d](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #8329:
URL: https://github.com/apache/pinot/pull/8329
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
walterddr commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829633997
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -255,6 +257,14 @@ public static URL makeKeyStoreUrl(String storePath)
return inputUri.toURL();
}
+ public static SSLContext getSslContext() {
+ return _sslContext != null ? _sslContext : SSLContexts.createDefault();
Review comment:
yeah. this is what I meant previously (less the `OVERRIDE` part which brilliantly solve the tls keystore init issue). thanks @richardstartin !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#issuecomment-1063683490
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8329](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0e789c4) into [master](https://codecov.io/gh/apache/pinot/commit/98d781092136eeb3a80e9e54fc8736f4c3d270f5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (98d7810) will **decrease** coverage by `33.35%`.
> The diff coverage is `38.86%`.
> :exclamation: Current head 0e789c4 differs from pull request most recent head 2b865bf. Consider uploading reports for the commit 2b865bf to get more accurate results
```diff
@@ Coverage Diff @@
## master #8329 +/- ##
=============================================
- Coverage 64.08% 30.73% -33.36%
=============================================
Files 1592 1627 +35
Lines 83916 85553 +1637
Branches 12706 12880 +174
=============================================
- Hits 53779 26291 -27488
- Misses 26261 56888 +30627
+ Partials 3876 2374 -1502
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `28.93% <35.62%> (?)` | |
| integration2 | `27.55% <34.41%> (?)` | |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...inot/controller/helix/ControllerRequestClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9Db250cm9sbGVyUmVxdWVzdENsaWVudC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ot/plugin/minion/tasks/SegmentConversionUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvU2VnbWVudENvbnZlcnNpb25VdGlscy5qYXZh) | `76.66% <ø> (+76.66%)` | :arrow_up: |
| [...he/pinot/segment/local/utils/SegmentPushUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9TZWdtZW50UHVzaFV0aWxzLmphdmE=) | `0.00% <0.00%> (-13.49%)` | :arrow_down: |
| [...org/apache/pinot/common/utils/http/HttpClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaHR0cC9IdHRwQ2xpZW50LmphdmE=) | `39.37% <39.37%> (ø)` | |
| [...n/java/org/apache/pinot/common/utils/TlsUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvVGxzVXRpbHMuamF2YQ==) | `76.40% <60.00%> (+76.40%)` | :arrow_up: |
| [...e/pinot/common/utils/FileUploadDownloadClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRmlsZVVwbG9hZERvd25sb2FkQ2xpZW50LmphdmE=) | `56.52% <61.29%> (+40.40%)` | :arrow_up: |
| [...ger/realtime/Server2ControllerSegmentUploader.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VydmVyMkNvbnRyb2xsZXJTZWdtZW50VXBsb2FkZXIuamF2YQ==) | `65.51% <100.00%> (-6.90%)` | :arrow_down: |
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | `58.09% <100.00%> (+51.42%)` | :arrow_up: |
| [.../tasks/BaseMultipleSegmentsConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZU11bHRpcGxlU2VnbWVudHNDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `90.00% <100.00%> (+86.66%)` | :arrow_up: |
| [...ion/tasks/BaseSingleSegmentConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVNpbmdsZVNlZ21lbnRDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `74.60% <100.00%> (+71.42%)` | :arrow_up: |
| ... and [1356 more](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [98d7810...2b865bf](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #8329: using HttpClient
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#issuecomment-1063683490
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8329](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (07d412a) into [master](https://codecov.io/gh/apache/pinot/commit/1a701b7d52c5d571689d4b2f8159f8c0cdda0acd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1a701b7) will **decrease** coverage by `41.88%`.
> The diff coverage is `56.77%`.
> :exclamation: Current head 07d412a differs from pull request most recent head 2e401e6. Consider uploading reports for the commit 2e401e6 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #8329 +/- ##
=============================================
- Coverage 70.86% 28.97% -41.89%
=============================================
Files 1635 1624 -11
Lines 85756 85417 -339
Branches 12912 12872 -40
=============================================
- Hits 60771 24751 -36020
- Misses 20793 58404 +37611
+ Partials 4192 2262 -1930
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `28.97% <56.77%> (-0.01%)` | :arrow_down: |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...inot/common/utils/fetcher/HttpsSegmentFetcher.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZmV0Y2hlci9IdHRwc1NlZ21lbnRGZXRjaGVyLmphdmE=) | `0.00% <0.00%> (-58.34%)` | :arrow_down: |
| [...he/pinot/segment/local/utils/SegmentPushUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9TZWdtZW50UHVzaFV0aWxzLmphdmE=) | `0.00% <0.00%> (-13.49%)` | :arrow_down: |
| [...e/pinot/common/utils/FileUploadDownloadClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRmlsZVVwbG9hZERvd25sb2FkQ2xpZW50LmphdmE=) | `50.00% <49.09%> (-13.74%)` | :arrow_down: |
| [...org/apache/pinot/common/utils/http/HttpClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaHR0cC9IdHRwQ2xpZW50LmphdmE=) | `58.44% <58.44%> (ø)` | |
| [...n/java/org/apache/pinot/common/utils/TlsUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvVGxzVXRpbHMuamF2YQ==) | `25.84% <60.00%> (-51.81%)` | :arrow_down: |
| [...ger/realtime/Server2ControllerSegmentUploader.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VydmVyMkNvbnRyb2xsZXJTZWdtZW50VXBsb2FkZXIuamF2YQ==) | `65.51% <100.00%> (-6.90%)` | :arrow_down: |
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | `58.49% <100.00%> (+0.39%)` | :arrow_up: |
| [.../tasks/BaseMultipleSegmentsConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZU11bHRpcGxlU2VnbWVudHNDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `90.00% <100.00%> (ø)` | |
| [...ion/tasks/BaseSingleSegmentConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVNpbmdsZVNlZ21lbnRDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `74.60% <100.00%> (ø)` | |
| [...ot/plugin/minion/tasks/SegmentConversionUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvU2VnbWVudENvbnZlcnNpb25VdGlscy5qYXZh) | `77.77% <100.00%> (+1.11%)` | :arrow_up: |
| ... and [1188 more](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [1a701b7...2e401e6](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#issuecomment-1063683490
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kkrugler commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
kkrugler commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829494766
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -60,6 +61,8 @@
private static final String TRUSTSTORE_PASSWORD = "truststore.password";
private static final String SSL_PROVIDER = "ssl.provider";
+ private static SSLContext _sslContext = null;
Review comment:
Just curious about the naming convention for static variables - I'd thought they were always upper-cased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] yupeng9 commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r828189443
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,371 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
Review comment:
make `Authorization ` a constant
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,371 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ String controllerHost = null;
+ String controllerVersion = null;
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ }
+ if (controllerHost != null) {
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
Review comment:
shall this be extended to talk to server/broker?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #8329: using HttpClient
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#issuecomment-1063683490
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8329](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (da1ce32) into [master](https://codecov.io/gh/apache/pinot/commit/8ef4dbacfb42596590d08c7c6f14f88093dfe7d5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8ef4dba) will **decrease** coverage by `41.92%`.
> The diff coverage is `56.08%`.
> :exclamation: Current head da1ce32 differs from pull request most recent head 6f2d43d. Consider uploading reports for the commit 6f2d43d to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #8329 +/- ##
=============================================
- Coverage 70.80% 28.88% -41.93%
=============================================
Files 1636 1625 -11
Lines 85804 85463 -341
Branches 12920 12881 -39
=============================================
- Hits 60755 24687 -36068
- Misses 20858 58491 +37633
+ Partials 4191 2285 -1906
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `28.88% <56.08%> (+<0.01%)` | :arrow_up: |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ot/plugin/minion/tasks/SegmentConversionUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvU2VnbWVudENvbnZlcnNpb25VdGlscy5qYXZh) | `76.66% <ø> (ø)` | |
| [...he/pinot/segment/local/utils/SegmentPushUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9TZWdtZW50UHVzaFV0aWxzLmphdmE=) | `0.00% <0.00%> (-13.49%)` | :arrow_down: |
| [...n/java/org/apache/pinot/common/utils/TlsUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvVGxzVXRpbHMuamF2YQ==) | `23.59% <20.00%> (-54.06%)` | :arrow_down: |
| [...e/pinot/common/utils/FileUploadDownloadClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRmlsZVVwbG9hZERvd25sb2FkQ2xpZW50LmphdmE=) | `51.48% <54.54%> (-11.52%)` | :arrow_down: |
| [...org/apache/pinot/common/utils/http/HttpClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaHR0cC9IdHRwQ2xpZW50LmphdmE=) | `58.75% <58.75%> (ø)` | |
| [...ger/realtime/Server2ControllerSegmentUploader.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VydmVyMkNvbnRyb2xsZXJTZWdtZW50VXBsb2FkZXIuamF2YQ==) | `65.51% <100.00%> (-6.90%)` | :arrow_down: |
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | `51.42% <100.00%> (ø)` | |
| [.../tasks/BaseMultipleSegmentsConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZU11bHRpcGxlU2VnbWVudHNDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `90.00% <100.00%> (ø)` | |
| [...ion/tasks/BaseSingleSegmentConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVNpbmdsZVNlZ21lbnRDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `74.60% <100.00%> (ø)` | |
| [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [1185 more](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8ef4dba...6f2d43d](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
walterddr commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r828433324
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,371 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendGetRequest(URI uri)
Review comment:
since this is a generic HTTPClient. I think we can remove the `@Deprecated` annotation yes @Jackie-Jiang ^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829533085
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -255,6 +257,14 @@ public static URL makeKeyStoreUrl(String storePath)
return inputUri.toURL();
}
+ public static SSLContext getSslContext() {
+ return _sslContext != null ? _sslContext : SSLContexts.createDefault();
Review comment:
I suggest putting it in a holder class:
```java
private static final AtomicReference<SSLContext> OVERRIDE = new AtomicReference<>();
public static void registerOverride(SSLContext override) {
if (!OVERRIDE.compareAndSet(null, override)) {
// warn that something else beat the caller here
}
}
private static final class Holder {
static final SSLContext SSL_CONTEXT = OVERRIDE.get() == null ? SSLContexts.createDefault() : OVERRIDE.get();
}
```
And then reference it via `Holder.SSL_CONTEXT` everywhere. This has a few advantages:
* registration is thread safe
* mutable until first use
* synchronization, at most once initialisation guaranteed by the classloader
* after initialisation, the `SSLContext` is constant which can drive optimisations like constant folding, which may or may not be important
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ String controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ String controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
Review comment:
Yes, please do not use `String.format` - it's ridiculously expensive. Most of the time we use it, it is when something has gone wrong so I suppose it is ok but it should never be used on the happy path.
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ String controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ String controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
Review comment:
In this particular case, since there are 3 parameters a varargs array will be allocated for every log statement which will always be logged since it's at INFO, so it would be better just to concatenate the strings to minimise logging overhead.
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -255,6 +262,38 @@ public static URL makeKeyStoreUrl(String storePath)
return inputUri.toURL();
}
+ public static SSLContext getSslContext() {
+ return SSLContextHolder.SSL_CONTEXT;
+ }
+
+ public static void setSslContext(SSLContext sslContext) {
+ registerOverride(sslContext);
+ }
+
+ private static void registerOverride(SSLContext override) {
+ if (!SSL_CONTEXT_REF.compareAndSet(null, override)) {
+ LOGGER.warn("SSL Context has already been set.");
+ // warn that something else beat the caller here
+ }
+ }
Review comment:
oh I'd just inline this into `setSslContext`, `registerOverride` was just a name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
walterddr commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r828435386
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,371 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ String controllerHost = null;
+ String controllerVersion = null;
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ }
+ if (controllerHost != null) {
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
Review comment:
this is a logging only usage. only Controller response attaches special HTTP response header. but yes we can add others later if necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #8329: using HttpClient
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#issuecomment-1063683490
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8329](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e288a0b) into [master](https://codecov.io/gh/apache/pinot/commit/8ef4dbacfb42596590d08c7c6f14f88093dfe7d5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8ef4dba) will **decrease** coverage by `0.09%`.
> The diff coverage is `45.95%`.
[](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #8329 +/- ##
============================================
- Coverage 70.80% 70.70% -0.10%
+ Complexity 4258 4255 -3
============================================
Files 1636 1638 +2
Lines 85804 85899 +95
Branches 12920 12927 +7
============================================
- Hits 60755 60739 -16
- Misses 20858 20969 +111
Partials 4191 4191
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `28.87% <36.08%> (-0.01%)` | :arrow_down: |
| integration2 | `27.65% <34.78%> (+0.14%)` | :arrow_up: |
| unittests1 | `66.85% <22.04%> (-0.10%)` | :arrow_down: |
| unittests2 | `14.17% <3.40%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...not/ingestion/common/DefaultControllerRestApi.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vdjBfZGVwcmVjYXRlZC9waW5vdC1pbmdlc3Rpb24tY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9pbmdlc3Rpb24vY29tbW9uL0RlZmF1bHRDb250cm9sbGVyUmVzdEFwaS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ot/plugin/minion/tasks/SegmentConversionUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvU2VnbWVudENvbnZlcnNpb25VdGlscy5qYXZh) | `76.66% <ø> (ø)` | |
| [...he/pinot/segment/local/utils/SegmentPushUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9TZWdtZW50UHVzaFV0aWxzLmphdmE=) | `13.48% <0.00%> (ø)` | |
| [...inot/controller/helix/ControllerRequestClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9Db250cm9sbGVyUmVxdWVzdENsaWVudC5qYXZh) | `19.04% <19.04%> (ø)` | |
| [...org/apache/pinot/common/utils/http/HttpClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaHR0cC9IdHRwQ2xpZW50LmphdmE=) | `41.66% <41.66%> (ø)` | |
| [...n/java/org/apache/pinot/common/utils/TlsUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvVGxzVXRpbHMuamF2YQ==) | `76.40% <60.00%> (-1.25%)` | :arrow_down: |
| [...e/pinot/common/utils/FileUploadDownloadClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRmlsZVVwbG9hZERvd25sb2FkQ2xpZW50LmphdmE=) | `64.35% <76.36%> (+1.35%)` | :arrow_up: |
| [...ger/realtime/Server2ControllerSegmentUploader.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VydmVyMkNvbnRyb2xsZXJTZWdtZW50VXBsb2FkZXIuamF2YQ==) | `72.41% <100.00%> (ø)` | |
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | `51.42% <100.00%> (ø)` | |
| [.../tasks/BaseMultipleSegmentsConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZU11bHRpcGxlU2VnbWVudHNDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `90.00% <100.00%> (ø)` | |
| ... and [25 more](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8ef4dba...e288a0b](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #8329: using HttpClient
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#issuecomment-1063683490
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8329](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e288a0b) into [master](https://codecov.io/gh/apache/pinot/commit/8ef4dbacfb42596590d08c7c6f14f88093dfe7d5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8ef4dba) will **decrease** coverage by `6.84%`.
> The diff coverage is `20.85%`.
[](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #8329 +/- ##
============================================
- Coverage 70.80% 63.96% -6.85%
+ Complexity 4258 4255 -3
============================================
Files 1636 1593 -43
Lines 85804 84015 -1789
Branches 12920 12723 -197
============================================
- Hits 60755 53741 -7014
- Misses 20858 26403 +5545
+ Partials 4191 3871 -320
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `66.85% <22.04%> (-0.10%)` | :arrow_down: |
| unittests2 | `14.17% <3.40%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | `6.66% <0.00%> (-44.77%)` | :arrow_down: |
| [...not/ingestion/common/DefaultControllerRestApi.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vdjBfZGVwcmVjYXRlZC9waW5vdC1pbmdlc3Rpb24tY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9pbmdlc3Rpb24vY29tbW9uL0RlZmF1bHRDb250cm9sbGVyUmVzdEFwaS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../tasks/BaseMultipleSegmentsConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZU11bHRpcGxlU2VnbWVudHNDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `3.33% <0.00%> (-86.67%)` | :arrow_down: |
| [...ion/tasks/BaseSingleSegmentConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVNpbmdsZVNlZ21lbnRDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `3.17% <0.00%> (-71.43%)` | :arrow_down: |
| [...ot/plugin/minion/tasks/SegmentConversionUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvU2VnbWVudENvbnZlcnNpb25VdGlscy5qYXZh) | `0.00% <ø> (-76.67%)` | :arrow_down: |
| [...he/pinot/segment/local/utils/SegmentPushUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9TZWdtZW50UHVzaFV0aWxzLmphdmE=) | `13.48% <0.00%> (ø)` | |
| [...inot/controller/helix/ControllerRequestClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9Db250cm9sbGVyUmVxdWVzdENsaWVudC5qYXZh) | `19.04% <19.04%> (ø)` | |
| [...n/java/org/apache/pinot/common/utils/TlsUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvVGxzVXRpbHMuamF2YQ==) | `1.12% <20.00%> (-76.53%)` | :arrow_down: |
| [...org/apache/pinot/common/utils/http/HttpClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaHR0cC9IdHRwQ2xpZW50LmphdmE=) | `20.00% <20.00%> (ø)` | |
| [...e/pinot/common/utils/FileUploadDownloadClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRmlsZVVwbG9hZERvd25sb2FkQ2xpZW50LmphdmE=) | `14.85% <27.27%> (-48.16%)` | :arrow_down: |
| ... and [385 more](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8ef4dba...e288a0b](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
walterddr commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829216994
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -255,6 +257,14 @@ public static URL makeKeyStoreUrl(String storePath)
return inputUri.toURL();
}
+ public static SSLContext getSslContext() {
+ return _sslContext != null ? _sslContext : SSLContexts.createDefault();
Review comment:
i dont know whether this needs to be synchronized as it is only generated once.
the only way to modify it is via setSslContext, which should also be statically accessed when the secure environment constructs at startup time).
CC the original author @apucher ^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829289735
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,371 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendGetRequest(URI uri)
Review comment:
Yes, we don't need to make it deprecated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829798759
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -255,6 +262,38 @@ public static URL makeKeyStoreUrl(String storePath)
return inputUri.toURL();
}
+ public static SSLContext getSslContext() {
+ return SSLContextHolder.SSL_CONTEXT;
+ }
+
+ public static void setSslContext(SSLContext sslContext) {
+ registerOverride(sslContext);
+ }
+
+ private static void registerOverride(SSLContext override) {
+ if (!SSL_CONTEXT_REF.compareAndSet(null, override)) {
+ LOGGER.warn("SSL Context has already been set.");
+ // warn that something else beat the caller here
+ }
+ }
Review comment:
oh I'd just inline this into `setSslContext`, `registerOverride` was just a name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829294078
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
Review comment:
This can be simplified to avoid duplicate code, same for other methods
```suggestion
return sendGetRequest(uri, null);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829535449
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ String controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ String controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
Review comment:
Yes, please do not use `String.format` - it's ridiculously expensive. Most of the time we use it, it is when something has gone wrong so I suppose it is ok but it should never be used on the happy path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829533085
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -255,6 +257,14 @@ public static URL makeKeyStoreUrl(String storePath)
return inputUri.toURL();
}
+ public static SSLContext getSslContext() {
+ return _sslContext != null ? _sslContext : SSLContexts.createDefault();
Review comment:
I suggest putting it in a holder class:
```java
private static final AtomicReference<SSLContext> OVERRIDE = new AtomicReference<>();
public static void registerOverride(SSLContext override) {
if (!OVERRIDE.compareAndSet(null, override)) {
// warn that something else beat the caller here
}
}
private static final class Holder {
static final SSLContext SSL_CONTEXT = OVERRIDE.get() == null ? SSLContexts.createDefault() : OVERRIDE.get();
}
```
And then reference it via `Holder.SSL_CONTEXT` everywhere. This has a few advantages:
* registration is thread safe
* mutable until first use
* synchronization, at most once initialisation guaranteed by the classloader
* after initialisation, the `SSLContext` is constant which can drive optimisations like constant folding, which may or may not be important
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
walterddr commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829638811
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
Review comment:
yeah weirdly we have some post method with empty body :-/
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ String controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ String controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
+ controllerVersion));
+ }
+ int statusCode = response.getStatusLine().getStatusCode();
+ if (statusCode >= 300) {
+ return new SimpleHttpResponse(statusCode, getErrorMessage(request, response));
+ }
+ return new SimpleHttpResponse(statusCode, EntityUtils.toString(response.getEntity()));
+ }
+ }
+
+ public CloseableHttpResponse execute(HttpUriRequest request)
+ throws IOException {
+ return _httpClient.execute(request);
+ }
+
+ // --------------------------------------------------------------------------
+ // Multi-part post/put APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendMultipartPostRequest(String url, String body)
+ throws IOException {
+ HttpPost post = new HttpPost(url);
+ // our handlers ignore key...so we can put anything here
+ MultipartEntityBuilder builder = MultipartEntityBuilder.create();
+ builder.addTextBody("body", body);
+ post.setEntity(builder.build());
Review comment:
for now i dont see it be useful. so i would prefer to add it when it is used.
(also tried to use sendPostRequest it had some weird test errors)
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
Review comment:
this would be a major refactoring. should we do this in a separate PR?
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -232,8 +235,7 @@ public static void installDefaultSSLSocketFactory(String keyStoreType, String ke
Protocol.registerProtocol("https",
new Protocol(CommonConstants.HTTPS_PROTOCOL, new PinotProtocolSocketFactory(sc.getSocketFactory()), 443));
- // FileUploadDownloadClient
- FileUploadDownloadClient.installDefaultSSLContext(sc);
+ _sslContext = sc;
Review comment:
yup. i guess it aligns with other comments will address it accordingly
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -60,6 +61,8 @@
private static final String TRUSTSTORE_PASSWORD = "truststore.password";
private static final String SSL_PROVIDER = "ssl.provider";
+ private static SSLContext _sslContext = null;
Review comment:
seems it only apply to private static final variables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
walterddr commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829216994
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -255,6 +257,14 @@ public static URL makeKeyStoreUrl(String storePath)
return inputUri.toURL();
}
+ public static SSLContext getSslContext() {
+ return _sslContext != null ? _sslContext : SSLContexts.createDefault();
Review comment:
i dont know whether this needs to be synchronized as it is only generated once.
the only way to modify it is via setSslContext, which should also be statically accessed when the secure environment constructs at startup time).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] yupeng9 commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r828188208
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,371 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendGetRequest(URI uri)
Review comment:
why do we add deprecated method, given this is a new file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
walterddr commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829216994
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -255,6 +257,14 @@ public static URL makeKeyStoreUrl(String storePath)
return inputUri.toURL();
}
+ public static SSLContext getSslContext() {
+ return _sslContext != null ? _sslContext : SSLContexts.createDefault();
Review comment:
i dont know whether this needs to be synchronized as it is only generated once.
the only way to modify it is via setSslContext, which should also be statically accessed when the secure environment constructs at startup time).
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -255,6 +257,14 @@ public static URL makeKeyStoreUrl(String storePath)
return inputUri.toURL();
}
+ public static SSLContext getSslContext() {
+ return _sslContext != null ? _sslContext : SSLContexts.createDefault();
Review comment:
i dont know whether this needs to be synchronized as it is only generated once.
the only way to modify it is via setSslContext, which should also be statically accessed when the secure environment constructs at startup time).
CC the original author @apucher ^
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -255,6 +257,14 @@ public static URL makeKeyStoreUrl(String storePath)
return inputUri.toURL();
}
+ public static SSLContext getSslContext() {
+ return _sslContext != null ? _sslContext : SSLContexts.createDefault();
Review comment:
yeah I was thinking about using lazy static initialization similar to https://stackoverflow.com/a/7420714 since it is static thread safe.
However the current way how TLSUtils install the TLSSpec is non-static. so it might not be possible to acquire the TLSSpecs at jvm start up time.
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -255,6 +257,14 @@ public static URL makeKeyStoreUrl(String storePath)
return inputUri.toURL();
}
+ public static SSLContext getSslContext() {
+ return _sslContext != null ? _sslContext : SSLContexts.createDefault();
Review comment:
yeah. this is what I meant previously (less the `OVERRIDE` part which brilliantly solve the tls keystore init issue). thanks @richardstartin !
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
Review comment:
yeah weirdly we have some post method with empty body :-/
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ String controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ String controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
+ controllerVersion));
+ }
+ int statusCode = response.getStatusLine().getStatusCode();
+ if (statusCode >= 300) {
+ return new SimpleHttpResponse(statusCode, getErrorMessage(request, response));
+ }
+ return new SimpleHttpResponse(statusCode, EntityUtils.toString(response.getEntity()));
+ }
+ }
+
+ public CloseableHttpResponse execute(HttpUriRequest request)
+ throws IOException {
+ return _httpClient.execute(request);
+ }
+
+ // --------------------------------------------------------------------------
+ // Multi-part post/put APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendMultipartPostRequest(String url, String body)
+ throws IOException {
+ HttpPost post = new HttpPost(url);
+ // our handlers ignore key...so we can put anything here
+ MultipartEntityBuilder builder = MultipartEntityBuilder.create();
+ builder.addTextBody("body", body);
+ post.setEntity(builder.build());
Review comment:
for now i dont see it be useful. so i would prefer to add it when it is used.
(also tried to use sendPostRequest it had some weird test errors)
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
Review comment:
this would be a major refactoring. should we do this in a separate PR?
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -232,8 +235,7 @@ public static void installDefaultSSLSocketFactory(String keyStoreType, String ke
Protocol.registerProtocol("https",
new Protocol(CommonConstants.HTTPS_PROTOCOL, new PinotProtocolSocketFactory(sc.getSocketFactory()), 443));
- // FileUploadDownloadClient
- FileUploadDownloadClient.installDefaultSSLContext(sc);
+ _sslContext = sc;
Review comment:
yup. i guess it aligns with other comments will address it accordingly
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -60,6 +61,8 @@
private static final String TRUSTSTORE_PASSWORD = "truststore.password";
private static final String SSL_PROVIDER = "ssl.provider";
+ private static SSLContext _sslContext = null;
Review comment:
seems it only apply to private static final variables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r830247106
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
Review comment:
Then we should use the regular post request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #8329: using HttpClient
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#issuecomment-1063683490
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8329](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5463ff4) into [master](https://codecov.io/gh/apache/pinot/commit/8ef4dbacfb42596590d08c7c6f14f88093dfe7d5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8ef4dba) will **decrease** coverage by `56.59%`.
> The diff coverage is `3.13%`.
> :exclamation: Current head 5463ff4 differs from pull request most recent head 5b585ee. Consider uploading reports for the commit 5b585ee to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #8329 +/- ##
=============================================
- Coverage 70.80% 14.20% -56.60%
+ Complexity 4258 81 -4177
=============================================
Files 1636 1593 -43
Lines 85804 84033 -1771
Branches 12920 12726 -194
=============================================
- Hits 60755 11939 -48816
- Misses 20858 71189 +50331
+ Partials 4191 905 -3286
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.20% <3.13%> (+0.02%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...e/pinot/common/utils/FileUploadDownloadClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRmlsZVVwbG9hZERvd25sb2FkQ2xpZW50LmphdmE=) | `0.00% <0.00%> (-63.01%)` | :arrow_down: |
| [...n/java/org/apache/pinot/common/utils/TlsUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvVGxzVXRpbHMuamF2YQ==) | `0.00% <0.00%> (-77.65%)` | :arrow_down: |
| [...org/apache/pinot/common/utils/http/HttpClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaHR0cC9IdHRwQ2xpZW50LmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...ger/realtime/Server2ControllerSegmentUploader.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VydmVyMkNvbnRyb2xsZXJTZWdtZW50VXBsb2FkZXIuamF2YQ==) | `0.00% <0.00%> (-72.42%)` | :arrow_down: |
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-51.43%)` | :arrow_down: |
| [...not/ingestion/common/DefaultControllerRestApi.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vdjBfZGVwcmVjYXRlZC9waW5vdC1pbmdlc3Rpb24tY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9pbmdlc3Rpb24vY29tbW9uL0RlZmF1bHRDb250cm9sbGVyUmVzdEFwaS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../tasks/BaseMultipleSegmentsConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZU11bHRpcGxlU2VnbWVudHNDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `3.33% <0.00%> (-86.67%)` | :arrow_down: |
| [...ion/tasks/BaseSingleSegmentConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVNpbmdsZVNlZ21lbnRDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `3.17% <0.00%> (-71.43%)` | :arrow_down: |
| [...ot/plugin/minion/tasks/SegmentConversionUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvU2VnbWVudENvbnZlcnNpb25VdGlscy5qYXZh) | `0.00% <ø> (-76.67%)` | :arrow_down: |
| [...he/pinot/segment/local/utils/SegmentPushUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9TZWdtZW50UHVzaFV0aWxzLmphdmE=) | `0.00% <0.00%> (-13.49%)` | :arrow_down: |
| ... and [1314 more](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8ef4dba...5b585ee](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kkrugler commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
kkrugler commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829485698
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ String controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ String controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
Review comment:
Just for my own edification, I thought that since we were using slf4j, the approved approach for formatted log msgs was to use slf4j's support for `{}` parameters. E.g.
``` java
LOGGER.info("Sending request: {} to controller: {}, version: {}", request.getURI(), controllerHost, controllerVersion);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829538546
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ String controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ String controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
Review comment:
In this particular case, since there are 3 parameters a varargs array will be allocated for every log statement which will always be logged since it's at INFO, so it would be better just to concatenate the strings to minimise logging overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829300127
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
Review comment:
Should `jsonRequestBody` be `Nullable`?
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ String controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ String controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
+ controllerVersion));
+ }
+ int statusCode = response.getStatusLine().getStatusCode();
+ if (statusCode >= 300) {
+ return new SimpleHttpResponse(statusCode, getErrorMessage(request, response));
+ }
+ return new SimpleHttpResponse(statusCode, EntityUtils.toString(response.getEntity()));
+ }
+ }
+
+ public CloseableHttpResponse execute(HttpUriRequest request)
+ throws IOException {
+ return _httpClient.execute(request);
+ }
+
+ // --------------------------------------------------------------------------
+ // Multi-part post/put APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendMultipartPostRequest(String url, String body)
+ throws IOException {
+ HttpPost post = new HttpPost(url);
+ // our handlers ignore key...so we can put anything here
+ MultipartEntityBuilder builder = MultipartEntityBuilder.create();
+ builder.addTextBody("body", body);
+ post.setEntity(builder.build());
+ try (CloseableHttpResponse response = _httpClient.execute(post)) {
+ StatusLine statusLine = response.getStatusLine();
+ int statusCode = statusLine.getStatusCode();
+ if (statusCode >= 300) {
+ return new SimpleHttpResponse(statusCode, getErrorMessage(post, response));
+ }
+ return new SimpleHttpResponse(statusCode, EntityUtils.toString(response.getEntity()));
+ }
+ }
+
+ public SimpleHttpResponse sendMultipartPutRequest(String url, String body)
+ throws IOException {
+ HttpPut put = new HttpPut(url);
+ // our handlers ignore key...so we can put anything here
+ MultipartEntityBuilder builder = MultipartEntityBuilder.create();
+ builder.addTextBody("body", body);
+ put.setEntity(builder.build());
+ try (CloseableHttpResponse response = _httpClient.execute(put)) {
+ StatusLine statusLine = response.getStatusLine();
+ int statusCode = statusLine.getStatusCode();
+ if (statusCode >= 300) {
+ return new SimpleHttpResponse(statusCode, getErrorMessage(put, response));
+ }
+ return new SimpleHttpResponse(statusCode, EntityUtils.toString(response.getEntity()));
+ }
+ }
+
+ // --------------------------------------------------------------------------
+ // Static utility for dealing with lower-level API responses.
+ // --------------------------------------------------------------------------
+
+ public static SimpleHttpResponse wrapAndThrowHttpException(SimpleHttpResponse resp)
+ throws HttpErrorStatusException {
+ if (resp.getStatusCode() >= 300) {
+ throw new HttpErrorStatusException(resp.getResponse(), resp.getStatusCode());
+ } else {
+ return resp;
+ }
+ }
+
+ public static String getErrorMessage(HttpUriRequest request, CloseableHttpResponse response) {
Review comment:
Make it `private`
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
Review comment:
Consider adding parameter for `headers`, `parameters`, `socketTimeoutMs` so that it is flexible. `sendGetRequest(uri)` will call `sendGetRequest(uri, null, null, null, DEFAULT_GET_REQUEST_TIMEOUT_MS)`. Same for other methods
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
Review comment:
Annotate `payload` and `headers` as `Nullable`, same for other methods
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
Review comment:
We should also include the `authToken`
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
Review comment:
Renaming to `sendJsonPostRequest` to be consistent? (e.g. we have `sendMultipartPostRequest`)
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ String controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ String controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
+ controllerVersion));
+ }
+ int statusCode = response.getStatusLine().getStatusCode();
+ if (statusCode >= 300) {
+ return new SimpleHttpResponse(statusCode, getErrorMessage(request, response));
+ }
+ return new SimpleHttpResponse(statusCode, EntityUtils.toString(response.getEntity()));
+ }
+ }
+
+ public CloseableHttpResponse execute(HttpUriRequest request)
Review comment:
Seems this is only used in `FileUploadDownloadClient.downloadFile()`, I'd suggest moving that method here as well. We should try to avoid exposing this API as caller needs to close the response
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ String controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ String controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
+ controllerVersion));
+ }
+ int statusCode = response.getStatusLine().getStatusCode();
+ if (statusCode >= 300) {
+ return new SimpleHttpResponse(statusCode, getErrorMessage(request, response));
+ }
+ return new SimpleHttpResponse(statusCode, EntityUtils.toString(response.getEntity()));
+ }
+ }
+
+ public CloseableHttpResponse execute(HttpUriRequest request)
+ throws IOException {
+ return _httpClient.execute(request);
+ }
+
+ // --------------------------------------------------------------------------
+ // Multi-part post/put APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendMultipartPostRequest(String url, String body)
+ throws IOException {
+ HttpPost post = new HttpPost(url);
+ // our handlers ignore key...so we can put anything here
+ MultipartEntityBuilder builder = MultipartEntityBuilder.create();
+ builder.addTextBody("body", body);
+ post.setEntity(builder.build());
Review comment:
Can we reuse `sendPostRequest`?
Also should we add `headers` and `authToken` support?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#issuecomment-1063683490
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8329](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (58eaef1) into [master](https://codecov.io/gh/apache/pinot/commit/98d781092136eeb3a80e9e54fc8736f4c3d270f5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (98d7810) will **decrease** coverage by `0.04%`.
> The diff coverage is `17.44%`.
```diff
@@ Coverage Diff @@
## master #8329 +/- ##
============================================
- Coverage 64.08% 64.04% -0.05%
- Complexity 4254 4267 +13
============================================
Files 1592 1596 +4
Lines 83916 84156 +240
Branches 12706 12730 +24
============================================
+ Hits 53779 53898 +119
- Misses 26261 26371 +110
- Partials 3876 3887 +11
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests1 | `66.94% <17.88%> (-0.02%)` | :arrow_down: |
| unittests2 | `14.22% <2.68%> (+0.02%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | `6.66% <0.00%> (ø)` | |
| [...not/ingestion/common/DefaultControllerRestApi.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vdjBfZGVwcmVjYXRlZC9waW5vdC1pbmdlc3Rpb24tY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9pbmdlc3Rpb24vY29tbW9uL0RlZmF1bHRDb250cm9sbGVyUmVzdEFwaS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../tasks/BaseMultipleSegmentsConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZU11bHRpcGxlU2VnbWVudHNDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `3.33% <0.00%> (ø)` | |
| [...ion/tasks/BaseSingleSegmentConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVNpbmdsZVNlZ21lbnRDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `3.17% <0.00%> (ø)` | |
| [...ot/plugin/minion/tasks/SegmentConversionUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvU2VnbWVudENvbnZlcnNpb25VdGlscy5qYXZh) | `0.00% <ø> (ø)` | |
| [...he/pinot/segment/local/utils/SegmentPushUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9TZWdtZW50UHVzaFV0aWxzLmphdmE=) | `13.48% <0.00%> (ø)` | |
| [...org/apache/pinot/common/utils/http/HttpClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaHR0cC9IdHRwQ2xpZW50LmphdmE=) | `14.46% <14.46%> (ø)` | |
| [...inot/controller/helix/ControllerRequestClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9Db250cm9sbGVyUmVxdWVzdENsaWVudC5qYXZh) | `17.77% <17.77%> (ø)` | |
| [...e/pinot/common/utils/FileUploadDownloadClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRmlsZVVwbG9hZERvd25sb2FkQ2xpZW50LmphdmE=) | `18.07% <22.85%> (+1.96%)` | :arrow_up: |
| [...n/java/org/apache/pinot/common/utils/TlsUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvVGxzVXRpbHMuamF2YQ==) | `4.21% <36.36%> (+4.21%)` | :arrow_up: |
| ... and [23 more](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [98d7810...58eaef1](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829288558
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -255,6 +257,14 @@ public static URL makeKeyStoreUrl(String storePath)
return inputUri.toURL();
}
+ public static SSLContext getSslContext() {
+ return _sslContext != null ? _sslContext : SSLContexts.createDefault();
Review comment:
Also CC @richardstartin to give suggestions on how to set static variables
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,371 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendGetRequest(URI uri)
Review comment:
Yes, we don't need to make it deprecated
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
Review comment:
This can be simplified to avoid duplicate code, same for other methods
```suggestion
return sendGetRequest(uri, null);
```
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
Review comment:
Should `jsonRequestBody` be `Nullable`?
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ String controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ String controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
+ controllerVersion));
+ }
+ int statusCode = response.getStatusLine().getStatusCode();
+ if (statusCode >= 300) {
+ return new SimpleHttpResponse(statusCode, getErrorMessage(request, response));
+ }
+ return new SimpleHttpResponse(statusCode, EntityUtils.toString(response.getEntity()));
+ }
+ }
+
+ public CloseableHttpResponse execute(HttpUriRequest request)
+ throws IOException {
+ return _httpClient.execute(request);
+ }
+
+ // --------------------------------------------------------------------------
+ // Multi-part post/put APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendMultipartPostRequest(String url, String body)
+ throws IOException {
+ HttpPost post = new HttpPost(url);
+ // our handlers ignore key...so we can put anything here
+ MultipartEntityBuilder builder = MultipartEntityBuilder.create();
+ builder.addTextBody("body", body);
+ post.setEntity(builder.build());
+ try (CloseableHttpResponse response = _httpClient.execute(post)) {
+ StatusLine statusLine = response.getStatusLine();
+ int statusCode = statusLine.getStatusCode();
+ if (statusCode >= 300) {
+ return new SimpleHttpResponse(statusCode, getErrorMessage(post, response));
+ }
+ return new SimpleHttpResponse(statusCode, EntityUtils.toString(response.getEntity()));
+ }
+ }
+
+ public SimpleHttpResponse sendMultipartPutRequest(String url, String body)
+ throws IOException {
+ HttpPut put = new HttpPut(url);
+ // our handlers ignore key...so we can put anything here
+ MultipartEntityBuilder builder = MultipartEntityBuilder.create();
+ builder.addTextBody("body", body);
+ put.setEntity(builder.build());
+ try (CloseableHttpResponse response = _httpClient.execute(put)) {
+ StatusLine statusLine = response.getStatusLine();
+ int statusCode = statusLine.getStatusCode();
+ if (statusCode >= 300) {
+ return new SimpleHttpResponse(statusCode, getErrorMessage(put, response));
+ }
+ return new SimpleHttpResponse(statusCode, EntityUtils.toString(response.getEntity()));
+ }
+ }
+
+ // --------------------------------------------------------------------------
+ // Static utility for dealing with lower-level API responses.
+ // --------------------------------------------------------------------------
+
+ public static SimpleHttpResponse wrapAndThrowHttpException(SimpleHttpResponse resp)
+ throws HttpErrorStatusException {
+ if (resp.getStatusCode() >= 300) {
+ throw new HttpErrorStatusException(resp.getResponse(), resp.getStatusCode());
+ } else {
+ return resp;
+ }
+ }
+
+ public static String getErrorMessage(HttpUriRequest request, CloseableHttpResponse response) {
Review comment:
Make it `private`
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
Review comment:
Consider adding parameter for `headers`, `parameters`, `socketTimeoutMs` so that it is flexible. `sendGetRequest(uri)` will call `sendGetRequest(uri, null, null, null, DEFAULT_GET_REQUEST_TIMEOUT_MS)`. Same for other methods
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
Review comment:
Annotate `payload` and `headers` as `Nullable`, same for other methods
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
Review comment:
We should also include the `authToken`
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
Review comment:
Renaming to `sendJsonPostRequest` to be consistent? (e.g. we have `sendMultipartPostRequest`)
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ String controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ String controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
+ controllerVersion));
+ }
+ int statusCode = response.getStatusLine().getStatusCode();
+ if (statusCode >= 300) {
+ return new SimpleHttpResponse(statusCode, getErrorMessage(request, response));
+ }
+ return new SimpleHttpResponse(statusCode, EntityUtils.toString(response.getEntity()));
+ }
+ }
+
+ public CloseableHttpResponse execute(HttpUriRequest request)
Review comment:
Seems this is only used in `FileUploadDownloadClient.downloadFile()`, I'd suggest moving that method here as well. We should try to avoid exposing this API as caller needs to close the response
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ String controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ String controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
+ controllerVersion));
+ }
+ int statusCode = response.getStatusLine().getStatusCode();
+ if (statusCode >= 300) {
+ return new SimpleHttpResponse(statusCode, getErrorMessage(request, response));
+ }
+ return new SimpleHttpResponse(statusCode, EntityUtils.toString(response.getEntity()));
+ }
+ }
+
+ public CloseableHttpResponse execute(HttpUriRequest request)
+ throws IOException {
+ return _httpClient.execute(request);
+ }
+
+ // --------------------------------------------------------------------------
+ // Multi-part post/put APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendMultipartPostRequest(String url, String body)
+ throws IOException {
+ HttpPost post = new HttpPost(url);
+ // our handlers ignore key...so we can put anything here
+ MultipartEntityBuilder builder = MultipartEntityBuilder.create();
+ builder.addTextBody("body", body);
+ post.setEntity(builder.build());
Review comment:
Can we reuse `sendPostRequest`?
Also should we add `headers` and `authToken` support?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kkrugler commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
kkrugler commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829485698
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,364 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String AUTH_HTTP_HEADER = "Authorization";
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader(AUTH_HTTP_HEADER, authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ String controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ String controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
Review comment:
Just for my own edification, I thought that since we were using slf4j, the approved approach for formatted log msgs was to use slf4j's support for `{}` parameters. E.g.
``` java
LOGGER.info("Sending request: {} to controller: {}, version: {}", request.getURI(), controllerHost, controllerVersion);
```
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -60,6 +61,8 @@
private static final String TRUSTSTORE_PASSWORD = "truststore.password";
private static final String SSL_PROVIDER = "ssl.provider";
+ private static SSLContext _sslContext = null;
Review comment:
Just curious about the naming convention for static variables - I'd thought they were always upper-cased.
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -232,8 +235,7 @@ public static void installDefaultSSLSocketFactory(String keyStoreType, String ke
Protocol.registerProtocol("https",
new Protocol(CommonConstants.HTTPS_PROTOCOL, new PinotProtocolSocketFactory(sc.getSocketFactory()), 443));
- // FileUploadDownloadClient
- FileUploadDownloadClient.installDefaultSSLContext(sc);
+ _sslContext = sc;
Review comment:
I think this should call `setSslContext()`, and that method should be synchronized, or use an `AtomicReference< SSLContext>`, to avoid potential concurrency issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #8329: using HttpClient
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#issuecomment-1063683490
# [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#8329](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a29ca6f) into [master](https://codecov.io/gh/apache/pinot/commit/8ef4dbacfb42596590d08c7c6f14f88093dfe7d5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8ef4dba) will **decrease** coverage by `56.63%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #8329 +/- ##
=============================================
- Coverage 70.80% 14.17% -56.64%
+ Complexity 4258 81 -4177
=============================================
Files 1636 1592 -44
Lines 85804 83932 -1872
Branches 12920 12714 -206
=============================================
- Hits 60755 11895 -48860
- Misses 20858 71144 +50286
+ Partials 4191 893 -3298
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.17% <0.00%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...e/pinot/common/utils/FileUploadDownloadClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRmlsZVVwbG9hZERvd25sb2FkQ2xpZW50LmphdmE=) | `0.00% <0.00%> (-63.01%)` | :arrow_down: |
| [...n/java/org/apache/pinot/common/utils/TlsUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvVGxzVXRpbHMuamF2YQ==) | `0.00% <0.00%> (-77.65%)` | :arrow_down: |
| [...org/apache/pinot/common/utils/http/HttpClient.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvaHR0cC9IdHRwQ2xpZW50LmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...ger/realtime/Server2ControllerSegmentUploader.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvU2VydmVyMkNvbnRyb2xsZXJTZWdtZW50VXBsb2FkZXIuamF2YQ==) | `0.00% <0.00%> (-72.42%)` | :arrow_down: |
| [...altime/ServerSegmentCompletionProtocolHandler.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL3JlYWx0aW1lL1NlcnZlclNlZ21lbnRDb21wbGV0aW9uUHJvdG9jb2xIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-51.43%)` | :arrow_down: |
| [...not/ingestion/common/DefaultControllerRestApi.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1iYXRjaC1pbmdlc3Rpb24vdjBfZGVwcmVjYXRlZC9waW5vdC1pbmdlc3Rpb24tY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9pbmdlc3Rpb24vY29tbW9uL0RlZmF1bHRDb250cm9sbGVyUmVzdEFwaS5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../tasks/BaseMultipleSegmentsConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZU11bHRpcGxlU2VnbWVudHNDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `3.33% <0.00%> (-86.67%)` | :arrow_down: |
| [...ion/tasks/BaseSingleSegmentConversionExecutor.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZVNpbmdsZVNlZ21lbnRDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `3.17% <0.00%> (-71.43%)` | :arrow_down: |
| [...ot/plugin/minion/tasks/SegmentConversionUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvU2VnbWVudENvbnZlcnNpb25VdGlscy5qYXZh) | `0.00% <ø> (-76.67%)` | :arrow_down: |
| [...he/pinot/segment/local/utils/SegmentPushUtils.java](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9TZWdtZW50UHVzaFV0aWxzLmphdmE=) | `0.00% <0.00%> (-13.49%)` | :arrow_down: |
| ... and [1313 more](https://codecov.io/gh/apache/pinot/pull/8329/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8ef4dba...a29ca6f](https://codecov.io/gh/apache/pinot/pull/8329?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
walterddr commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829305394
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -255,6 +257,14 @@ public static URL makeKeyStoreUrl(String storePath)
return inputUri.toURL();
}
+ public static SSLContext getSslContext() {
+ return _sslContext != null ? _sslContext : SSLContexts.createDefault();
Review comment:
yeah I was thinking about using lazy static initialization similar to https://stackoverflow.com/a/7420714 since it is static thread safe.
However the current way how TLSUtils install the TLSSpec is non-static. so it might not be possible to acquire the TLSSpecs at jvm start up time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kkrugler commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
kkrugler commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829506216
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -232,8 +235,7 @@ public static void installDefaultSSLSocketFactory(String keyStoreType, String ke
Protocol.registerProtocol("https",
new Protocol(CommonConstants.HTTPS_PROTOCOL, new PinotProtocolSocketFactory(sc.getSocketFactory()), 443));
- // FileUploadDownloadClient
- FileUploadDownloadClient.installDefaultSSLContext(sc);
+ _sslContext = sc;
Review comment:
I think this should call `setSslContext()`, and that method should be synchronized, or use an `AtomicReference< SSLContext>`, to avoid potential concurrency issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] yupeng9 commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r828194687
##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/ControllerRequestClient.java
##########
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.net.URL;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.http.HttpClient;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.tenant.Tenant;
+import org.apache.pinot.spi.config.tenant.TenantRole;
+import org.apache.pinot.spi.data.Schema;
+
+
+public class ControllerRequestClient {
Review comment:
need docs
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -255,6 +257,14 @@ public static URL makeKeyStoreUrl(String storePath)
return inputUri.toURL();
}
+ public static SSLContext getSslContext() {
+ return _sslContext != null ? _sslContext : SSLContexts.createDefault();
Review comment:
synchronized?
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,371 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
Review comment:
make `Authorization ` a constant
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/http/HttpClient.java
##########
@@ -0,0 +1,371 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils.http;
+
+import java.io.IOException;
+import java.net.URI;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import javax.net.ssl.SSLContext;
+import org.apache.commons.collections.MapUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.Header;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHeaders;
+import org.apache.http.HttpVersion;
+import org.apache.http.NameValuePair;
+import org.apache.http.StatusLine;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.client.methods.HttpUriRequest;
+import org.apache.http.client.methods.RequestBuilder;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.entity.ContentType;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.entity.mime.MultipartEntityBuilder;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.message.BasicHeader;
+import org.apache.http.util.EntityUtils;
+import org.apache.pinot.common.exception.HttpErrorStatusException;
+import org.apache.pinot.common.utils.SimpleHttpErrorInfo;
+import org.apache.pinot.common.utils.SimpleHttpResponse;
+import org.apache.pinot.common.utils.TlsUtils;
+import org.apache.pinot.spi.utils.CommonConstants;
+import org.apache.pinot.spi.utils.JsonUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * The {@code HTTPClient} wraps around a {@link CloseableHttpClient} to provide a reusable client for making
+ * HTTP requests.
+ */
+public class HttpClient implements AutoCloseable {
+ private static final Logger LOGGER = LoggerFactory.getLogger(HttpClient.class);
+
+ public static final int DEFAULT_SOCKET_TIMEOUT_MS = 600 * 1000; // 10 minutes
+ public static final int GET_REQUEST_SOCKET_TIMEOUT_MS = 5 * 1000; // 5 seconds
+ public static final int DELETE_REQUEST_SOCKET_TIMEOUT_MS = 10 * 1000; // 10 seconds
+ public static final String JSON_CONTENT_TYPE = "application/json";
+
+ private final CloseableHttpClient _httpClient;
+
+ public HttpClient() {
+ this(null);
+ }
+
+ public HttpClient(@Nullable SSLContext sslContext) {
+ SSLContext context = sslContext != null ? sslContext : TlsUtils.getSslContext();
+ // Set NoopHostnameVerifier to skip validating hostname when uploading/downloading segments.
+ SSLConnectionSocketFactory csf = new SSLConnectionSocketFactory(context, NoopHostnameVerifier.INSTANCE);
+ _httpClient = HttpClients.custom().setSSLSocketFactory(csf).build();
+ }
+
+ // --------------------------------------------------------------------------
+ // Generic HTTP Request APIs
+ // --------------------------------------------------------------------------
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendGetRequest(URI, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendGetRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendGetRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.get(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
+ }
+ setTimeout(requestBuilder, GET_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendDeleteRequest(URI, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendDeleteRequest(URI uri)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ public SimpleHttpResponse sendDeleteRequest(URI uri, @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.delete(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPostRequest(URI, HttpEntity, Map, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPostRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPostRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.post(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
+ }
+ if (MapUtils.isNotEmpty(headers)) {
+ for (Map.Entry<String, String> header : headers.entrySet()) {
+ requestBuilder.addHeader(header.getKey(), header.getValue());
+ }
+ }
+ setTimeout(requestBuilder, DEFAULT_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ /**
+ * Deprecated due to lack of auth header support. May break for deployments with auth enabled
+ *
+ * @see #sendPutRequest(URI, HttpEntity, Map, String)
+ */
+ @Deprecated
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers)
+ throws IOException {
+ return sendPutRequest(uri, payload, headers, null);
+ }
+
+ public SimpleHttpResponse sendPutRequest(URI uri, HttpEntity payload, Map<String, String> headers,
+ @Nullable String authToken)
+ throws IOException {
+ RequestBuilder requestBuilder = RequestBuilder.put(uri).setVersion(HttpVersion.HTTP_1_1);
+ if (payload != null) {
+ requestBuilder.setEntity(payload);
+ }
+ if (StringUtils.isNotBlank(authToken)) {
+ requestBuilder.addHeader("Authorization", authToken);
+ }
+ setTimeout(requestBuilder, DELETE_REQUEST_SOCKET_TIMEOUT_MS);
+ return sendRequest(requestBuilder.build());
+ }
+
+ // --------------------------------------------------------------------------
+ // JSON post/put utility APIs
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return postJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse postJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPostRequest(uri, entity, headers, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String jsonRequestBody)
+ throws IOException {
+ return putJsonRequest(uri, jsonRequestBody, null);
+ }
+
+ public SimpleHttpResponse putJsonRequest(URI uri, @Nullable String requestBody,
+ @Nullable Map<String, String> headers)
+ throws IOException {
+ if (MapUtils.isEmpty(headers)) {
+ headers = new HashMap<>();
+ }
+ headers.put(HttpHeaders.CONTENT_TYPE, JSON_CONTENT_TYPE);
+ HttpEntity entity = requestBody == null ? null : new StringEntity(requestBody, ContentType.APPLICATION_JSON);
+ return sendPutRequest(uri, entity, headers, null);
+ }
+
+ // --------------------------------------------------------------------------
+ // Lower-level request/execute APIs.
+ // --------------------------------------------------------------------------
+
+ public SimpleHttpResponse sendRequest(HttpUriRequest request)
+ throws IOException {
+ try (CloseableHttpResponse response = _httpClient.execute(request)) {
+ String controllerHost = null;
+ String controllerVersion = null;
+ if (response.containsHeader(CommonConstants.Controller.HOST_HTTP_HEADER)) {
+ controllerHost = response.getFirstHeader(CommonConstants.Controller.HOST_HTTP_HEADER).getValue();
+ controllerVersion = response.getFirstHeader(CommonConstants.Controller.VERSION_HTTP_HEADER).getValue();
+ }
+ if (controllerHost != null) {
+ LOGGER.info(String
+ .format("Sending request: %s to controller: %s, version: %s", request.getURI(), controllerHost,
Review comment:
shall this be extended to talk to server/broker?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #8329: Refactor Pinot http utils into HttpClient
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #8329:
URL: https://github.com/apache/pinot/pull/8329#discussion_r829288558
##########
File path: pinot-common/src/main/java/org/apache/pinot/common/utils/TlsUtils.java
##########
@@ -255,6 +257,14 @@ public static URL makeKeyStoreUrl(String storePath)
return inputUri.toURL();
}
+ public static SSLContext getSslContext() {
+ return _sslContext != null ? _sslContext : SSLContexts.createDefault();
Review comment:
Also CC @richardstartin to give suggestions on how to set static variables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org