You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "nsivarajan (via GitHub)" <gi...@apache.org> on 2026/02/28 19:13:16 UTC
[PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
nsivarajan opened a new pull request, #60921:
URL: https://github.com/apache/doris/pull/60921
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Currently, Doris provides an enable_https switch to enforce HTTPS connections. In hardened deployments, HTTP is completely disabled by setting http_port = 0. However, intra FE-to-FE communication still relies on HTTP, causing failures in edit log and checkpoint synchronisation when HTTPS is enabled.
This PR enhances HTTPS support by automatically converting intra FE-to-FE communication to HTTPS when enable_https=true, without introducing any new configuration, leveraging mysql_ssl_default_ca_certificate created for https connection. When enable_https=false, behavior remains unchanged. This ensures secure and seamless FE-to-FE communication without breaking existing workflows.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [X] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [X] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-3988639762
# FE UT Coverage Report
Increment line coverage ` 8.33% (6/72)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60921_29f45dbe504c8d46be0de681c14f24d3a97c8291/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60921_29f45dbe504c8d46be0de681c14f24d3a97c8291/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4632644184
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r3363551053
##########
fe/fe-core/src/main/java/org/apache/doris/master/MetaHelper.java:
##########
@@ -149,9 +149,13 @@ private static void checkFile(File file) throws IOException {
public static <T> ResponseBody doGet(String url, int timeout, Class<T> clazz) throws IOException {
Map<String, String> headers = HttpURLUtil.getNodeIdentHeaders();
- LOG.info("meta helper, url: {}, timeout{}, headers: {}", url, timeout, headers);
+ LOG.debug("meta helper, url: {}, timeout: {}, headers: {}", url, timeout, headers);
String response = HttpUtils.doGet(url, headers, timeout);
Review Comment:
HTTP request logs are for audit purposes — don't guard them with debugEnable. Use log.Info instead.↳
For debug logs, always wrap with if debugEnable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r3389415232
##########
fe/fe-core/src/main/java/org/apache/doris/httpv2/meta/MetaService.java:
##########
@@ -163,8 +163,7 @@ public Object put(HttpServletRequest request, HttpServletResponse response) thro
clientHost, clientPort, machine, portStr);
clientHost = Strings.isNullOrEmpty(clientHost) ? machine : clientHost;
- String url = "http://" + NetUtils.getHostPortInAccessibleFormat(clientHost, Integer.valueOf(portStr))
- + "/image?version=" + versionStr;
+ String url = HttpURLUtil.buildInternalFeUrl(clientHost, "/image", "version=" + versionStr);
Review Comment:
All FEs in a Doris cluster are expected to share the same port configuration, so in practice both approaches yield identical results. However, the scenario you described — ports differing across nodes during a rolling config change — is rare but valid. Fixed in the latest commit by using the already-validated portStr directly with the correct scheme, preserving the original design contract.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r3339455114
##########
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/manager/HttpUtils.java:
##########
@@ -60,18 +61,21 @@ public class HttpUtils {
static final int DEFAULT_TIME_OUT_MS = 2000;
public static List<Pair<String, Integer>> getFeList() {
+ int port = Config.enable_https ? Config.https_port : Config.http_port;
return Env.getCurrentEnv().getFrontends(null)
- .stream().filter(Frontend::isAlive).map(fe -> Pair.of(fe.getHost(), Config.http_port))
+ .stream().filter(Frontend::isAlive).map(fe -> Pair.of(fe.getHost(), port))
.collect(Collectors.toList());
}
static boolean isCurrentFe(String ip, int port) {
Review Comment:
This method is not used?
##########
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/manager/QueryProfileAction.java:
##########
@@ -196,7 +196,7 @@ public Object queryInfo(HttpServletRequest request, HttpServletResponse response
// add node information
for (List<String> query : queries) {
query.set(1, NetUtils.getHostPortInAccessibleFormat(Env.getCurrentEnv().getSelfNode().getHost(),
- Config.http_port));
+ Config.enable_https ? Config.https_port : Config.http_port));
Review Comment:
### Remove the repeated port ternary
`Config.enable_https ? Config.https_port : Config.http_port` is repeated about ten times across `NodeAction`, `ClusterAction`, `QueryProfileAction`, `HeartbeatMgr`, `HttpUtils`, and `MinidumpUtils`. The PR already centralizes URL construction in `buildInternalFeUrl`; do the same for the bare port selection (e.g. a `HttpURLUtil.getInternalFePort()` helper or a convenience accessor on `Config`). This removes scattered copies of the same branch and reduces the risk of a future partial change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4601413931
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r3421495765
##########
fe/fe-core/src/main/java/org/apache/doris/common/util/InternalHttpsUtils.java:
##########
@@ -0,0 +1,98 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.util;
+
+import org.apache.doris.common.Config;
+
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.security.KeyStore;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+
+/**
+ * Utility for creating SSL-aware HTTP clients for internal FE-to-FE communication.
+ *
+ * <p>Builds an {@link SSLContext} from the configured CA truststore once and caches it.
+ * Hostname verification is disabled for IP-based intra-cluster connections.
+ * Certificate rotation requires a FE restart.
+ */
+public class InternalHttpsUtils {
+ private static volatile SSLContext cachedSslContext = null;
+
+ private static final Logger LOG = LogManager.getLogger(InternalHttpsUtils.class);
+
+ /**
+ * Returns the cached SSLContext, building it from the configured truststore on first call.
+ */
+ public static SSLContext getSslContext() {
+ if (cachedSslContext == null) {
+ synchronized (InternalHttpsUtils.class) {
+ if (cachedSslContext == null) {
+ cachedSslContext = buildSslContext();
+ }
+ }
+ }
+ return cachedSslContext;
+ }
+
+ private static SSLContext buildSslContext() {
+ try {
+ KeyStore trustStore = KeyStore.getInstance(Config.ssl_trust_store_type);
+ try (InputStream stream = Files.newInputStream(
+ Paths.get(Config.mysql_ssl_default_ca_certificate))) {
+ trustStore.load(stream, Config.mysql_ssl_default_ca_certificate_password.toCharArray());
Review Comment:
This truststore is independent from the certificate that the FE HTTPS server actually presents. `HttpServer` is configured from `Config.key_store_path`, `Config.key_store_type`, and `Config.key_store_password`, while this client trusts `Config.mysql_ssl_default_ca_certificate` with `Config.ssl_trust_store_type`. An existing HTTPS deployment can therefore have a valid FE `key_store_path` certificate that is not signed by the MySQL SSL CA, causing the new internal HTTPS calls (`/check`, `/role`, checkpoint image sync, query/profile fan-out) to fail PKIX validation. Please add/use a FE HTTPS truststore config that matches the server certificate chain, or derive this context from the configured FE HTTPS certificate/CA, and cover it with an end-to-end HTTPS client/server test.
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##########
@@ -1823,7 +1822,7 @@ private void transferToMaster() {
toMasterProgress = "log master info";
this.masterInfo = new MasterInfo(Env.getCurrentEnv().getSelfNode().getHost(),
- Config.http_port,
+ HttpURLUtil.getHttpPort(),
Config.rpc_port);
Review Comment:
Changing `MasterInfo` to store `https_port` makes every `Env.getMasterHttpPort()` consumer point at the HTTPS listener. One important consumer is `RestBaseController.forwardToMaster()` (used by APIs such as `/api/{db}/_cancel` and several `/rest/v2/...` manager calls): it builds the master URL from `request.getScheme()` plus `env.getMasterHttpPort()` and sends it with a vanilla `new RestTemplate()`. With `enable_https=true`, after the HTTP-to-HTTPS redirect that becomes `https://master:<https_port>/...`, but that `RestTemplate` does not use `InternalHttpsUtils`' truststore or hostname-verifier handling, so deployments using Doris/internal self-signed FE certificates fail with `SSLHandshakeException` instead of forwarding to the master. Please update the generic forward-to-master path to use the same HTTPS-aware trust configuration, or avoid switching `MasterInfo` to the HTTPS port until all its consumers are scheme/TLS aware.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4726474888
# FE Regression Coverage Report
Increment line coverage ` 14.62% (19/130)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60921_affe1d2a5247d28e273bbc3132b34306e9142694_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60921_affe1d2a5247d28e273bbc3132b34306e9142694_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-3982791245
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-3988546452
<details>
<summary>TPC-H: <b>Total hot run time: 29084 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 29f45dbe504c8d46be0de681c14f24d3a97c8291, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17606 4404 4317 4317
q2 q3 10651 792 533 533
q4 4675 369 255 255
q5 7553 1220 1009 1009
q6 176 178 149 149
q7 784 839 678 678
q8 9743 1467 1330 1330
q9 5243 4798 4744 4744
q10 6825 1896 1648 1648
q11 452 255 260 255
q12 763 576 482 482
q13 17767 4242 3421 3421
q14 243 232 213 213
q15 957 794 793 793
q16 774 739 680 680
q17 730 889 412 412
q18 5983 5317 5312 5312
q19 1466 982 618 618
q20 526 511 413 413
q21 4681 1982 1556 1556
q22 408 337 266 266
Total cold run time: 98006 ms
Total hot run time: 29084 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4725 4574 4697 4574
q2 q3 1819 2229 1779 1779
q4 861 1189 785 785
q5 4055 4394 4364 4364
q6 183 172 139 139
q7 1757 1629 1523 1523
q8 2490 2708 2625 2625
q9 7429 7430 7381 7381
q10 2630 2833 2400 2400
q11 526 433 408 408
q12 507 613 465 465
q13 4042 4584 3667 3667
q14 339 305 277 277
q15 835 799 810 799
q16 719 778 737 737
q17 1175 1564 1392 1392
q18 7295 6767 6768 6767
q19 940 1081 1007 1007
q20 2088 2196 2008 2008
q21 4092 3511 3473 3473
q22 531 448 419 419
Total cold run time: 49038 ms
Total hot run time: 46989 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r2882559768
##########
fe/fe-core/src/main/java/org/apache/doris/master/MetaHelper.java:
##########
@@ -39,7 +39,6 @@
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
-import java.util.Map;
public class MetaHelper {
public static final Logger LOG = LogManager.getLogger(MetaHelper.class);
Review Comment:
Restored debug with appropriate info
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-3985576927
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4060787998
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4587339782
# FE UT Coverage Report
Increment line coverage ` 21.95% (18/82)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60921_5c1f94c0bd3128aced58ce172868708101d080d7/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60921_5c1f94c0bd3128aced58ce172868708101d080d7/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4586635852
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4719379781
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4725913044
<details>
<summary>TPC-DS: <b>Total hot run time: 175154 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit affe1d2a5247d28e273bbc3132b34306e9142694, data reload: false
query5 4312 626 493 493
query6 437 194 176 176
query7 4810 541 296 296
query8 362 220 207 207
query9 8759 4124 4123 4123
query10 433 303 250 250
query11 5927 2358 2107 2107
query12 156 101 100 100
query13 1294 594 426 426
query14 6379 5383 5078 5078
query14_1 4392 4410 4402 4402
query15 207 193 173 173
query16 1049 459 424 424
query17 1109 684 557 557
query18 2462 468 336 336
query19 217 178 136 136
query20 110 106 102 102
query21 214 136 114 114
query22 13722 13590 13410 13410
query23 17231 16451 16098 16098
query23_1 16293 16191 16179 16179
query24 7619 1782 1310 1310
query24_1 1330 1334 1329 1329
query25 581 475 388 388
query26 1311 328 177 177
query27 2669 572 342 342
query28 4458 2062 2074 2062
query29 1085 624 505 505
query30 313 244 198 198
query31 1144 1087 966 966
query32 105 66 60 60
query33 609 322 261 261
query34 1161 1154 635 635
query35 760 831 697 697
query36 1402 1378 1226 1226
query37 155 105 91 91
query38 3214 3129 3035 3035
query39 930 955 905 905
query39_1 882 919 866 866
query40 226 126 103 103
query41 69 68 69 68
query42 97 94 93 93
query43 332 332 283 283
query44 1459 788 786 786
query45 200 190 179 179
query46 1068 1195 771 771
query47 2364 2331 2187 2187
query48 412 455 309 309
query49 618 482 337 337
query50 941 357 262 262
query51 4331 4370 4207 4207
query52 88 88 75 75
query53 252 269 179 179
query54 264 205 216 205
query55 77 75 69 69
query56 236 220 205 205
query57 1433 1428 1349 1349
query58 234 235 207 207
query59 1574 1667 1436 1436
query60 281 237 226 226
query61 153 149 145 145
query62 695 652 590 590
query63 228 185 199 185
query64 2506 759 607 607
query65 4839 4819 4787 4787
query66 1769 460 334 334
query67 29758 29671 29460 29460
query68 3221 1441 972 972
query69 417 297 266 266
query70 1089 979 904 904
query71 298 231 207 207
query72 2937 2610 2315 2315
query73 842 835 462 462
query74 5130 4955 4738 4738
query75 2617 2594 2250 2250
query76 2312 1206 800 800
query77 350 371 289 289
query78 12565 12424 11855 11855
query79 1412 1206 812 812
query80 1281 474 376 376
query81 509 276 238 238
query82 610 160 122 122
query83 329 277 252 252
query84 257 146 111 111
query85 926 498 421 421
query86 430 312 281 281
query87 3393 3345 3155 3155
query88 3718 2854 2814 2814
query89 427 391 335 335
query90 1875 178 176 176
query91 172 161 131 131
query92 63 65 59 59
query93 1609 1470 876 876
query94 740 344 304 304
query95 694 360 344 344
query96 1076 798 364 364
query97 2731 2697 2579 2579
query98 216 208 209 208
query99 1169 1170 1015 1015
Total cold run time: 261940 ms
Total hot run time: 175154 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-3986083447
<details>
<summary>TPC-DS: <b>Total hot run time: 183423 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 29f45dbe504c8d46be0de681c14f24d3a97c8291, data reload: false
query5 4329 646 528 528
query6 324 213 197 197
query7 4207 462 270 270
query8 334 246 234 234
query9 8698 2764 2751 2751
query10 530 385 340 340
query11 16970 17297 17154 17154
query12 195 147 136 136
query13 1419 479 404 404
query14 6572 3408 3105 3105
query14_1 2910 2895 2898 2895
query15 207 203 176 176
query16 1001 491 473 473
query17 1425 731 640 640
query18 2524 451 356 356
query19 204 203 189 189
query20 150 133 132 132
query21 264 161 119 119
query22 5631 4998 4903 4903
query23 17149 16805 16500 16500
query23_1 16681 16661 16546 16546
query24 7280 1626 1240 1240
query24_1 1259 1257 1216 1216
query25 566 489 426 426
query26 1244 258 162 162
query27 2770 469 289 289
query28 4534 1897 1903 1897
query29 809 578 477 477
query30 318 247 207 207
query31 883 729 656 656
query32 81 73 71 71
query33 525 356 295 295
query34 945 928 564 564
query35 664 695 623 623
query36 1111 1127 929 929
query37 143 95 89 89
query38 2963 2886 2840 2840
query39 884 871 856 856
query39_1 838 836 836 836
query40 240 165 138 138
query41 68 64 63 63
query42 107 102 99 99
query43 376 393 349 349
query44
query45 206 199 188 188
query46 884 991 612 612
query47 2105 2147 2036 2036
query48 322 345 245 245
query49 642 473 393 393
query50 697 294 220 220
query51 4086 4103 4062 4062
query52 107 111 98 98
query53 291 332 288 288
query54 312 280 301 280
query55 85 88 82 82
query56 349 323 338 323
query57 1375 1344 1250 1250
query58 297 323 271 271
query59 2554 2686 2487 2487
query60 320 336 315 315
query61 145 146 144 144
query62 631 582 547 547
query63 309 282 273 273
query64 4881 1251 965 965
query65
query66 1453 459 351 351
query67 16657 16396 16348 16348
query68
query69 394 306 282 282
query70 969 950 979 950
query71 343 306 291 291
query72 2797 2649 2430 2430
query73 543 548 318 318
query74 9998 9947 9763 9763
query75 2842 2746 2429 2429
query76 2290 1024 676 676
query77 366 368 304 304
query78 11093 11278 10626 10626
query79 2353 796 602 602
query80 1755 628 543 543
query81 548 274 242 242
query82 1015 146 114 114
query83 322 283 244 244
query84 268 111 101 101
query85 883 474 427 427
query86 411 299 303 299
query87 3097 3115 2953 2953
query88 3552 2683 2677 2677
query89 419 366 340 340
query90 1998 172 169 169
query91 164 159 137 137
query92 77 75 67 67
query93 991 846 507 507
query94 624 307 299 299
query95 585 403 305 305
query96 641 511 229 229
query97 2479 2481 2422 2422
query98 226 221 217 217
query99 1006 1015 923 923
Total cold run time: 255369 ms
Total hot run time: 183423 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r2876970755
##########
fe/fe-core/src/main/java/org/apache/doris/common/util/InternalHttpsUtils.java:
##########
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.util;
+
+import org.apache.doris.common.Config;
+
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.security.KeyStore;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+
+/**
+ * SSL-aware HTTP clients for internal FE communication using MySQL SSL truststore.
+ *
+ * Security Model:
+ * - Validates certificates against configured CA truststore (mysql_ssl_default_ca_certificate)
+ * - Hostname verification is DISABLED to support IP-based FE communication
+ * - This is safe for internal cluster communication because:
+ * 1. All endpoints enforce checkFromValidFe() - only registered FE nodes can connect
+ * 2. FE cluster is assumed to be on trusted network
+ * 3. Traffic is encrypted and authenticated via certificate validation
+ *
+ * This approach is similar to other distributed systems (Kafka, Elasticsearch, Cassandra)
+ * where inter-node SSL communication disables hostname verification for operational flexibility.
+ */
+public class InternalHttpsUtils {
+ private static final Logger LOG = LogManager.getLogger(InternalHttpsUtils.class);
+
+ public static CloseableHttpClient createValidatedHttpClient() {
+ try {
+ KeyStore trustStore = KeyStore.getInstance(Config.ssl_trust_store_type);
+ try (InputStream stream = Files.newInputStream(
+ Paths.get(Config.mysql_ssl_default_ca_certificate))) {
+ trustStore.load(stream, Config.mysql_ssl_default_ca_certificate_password.toCharArray());
+ }
+
+ TrustManagerFactory tmf = TrustManagerFactory.getInstance(
+ TrustManagerFactory.getDefaultAlgorithm());
+ tmf.init(trustStore);
+
+ SSLContext sslContext = SSLContext.getInstance("TLS");
+ sslContext.init(null, tmf.getTrustManagers(), null);
+
+ SSLConnectionSocketFactory sslFactory = new SSLConnectionSocketFactory(
+ sslContext,
+ NoopHostnameVerifier.INSTANCE);
+
+ return HttpClients.custom()
+ .setSSLSocketFactory(sslFactory)
+ .build();
+ } catch (Exception e) {
+ LOG.error("Failed to create SSL-aware HTTP client using truststore: {}",
+ Config.mysql_ssl_default_ca_certificate, e);
+ throw new RuntimeException("Failed to create SSL-aware HTTP client", e);
+ }
+ }
+
+ public static void installTrustManagerForUrlConnection() {
+ try {
+ KeyStore trustStore = KeyStore.getInstance(Config.ssl_trust_store_type);
+ try (InputStream stream = Files.newInputStream(
+ Paths.get(Config.mysql_ssl_default_ca_certificate))) {
+ trustStore.load(stream, Config.mysql_ssl_default_ca_certificate_password.toCharArray());
+ }
+
+ TrustManagerFactory tmf = TrustManagerFactory.getInstance(
+ TrustManagerFactory.getDefaultAlgorithm());
+ tmf.init(trustStore);
+
+ SSLContext sslContext = SSLContext.getInstance("TLS");
+ sslContext.init(null, tmf.getTrustManagers(), null);
+
+ javax.net.ssl.HttpsURLConnection.setDefaultSSLSocketFactory(sslContext.getSocketFactory());
+ javax.net.ssl.HttpsURLConnection.setDefaultHostnameVerifier((hostname, session) -> true);
Review Comment:
**Critical: JVM-wide global state mutation**
`setDefaultSSLSocketFactory()` and `setDefaultHostnameVerifier()` modify **JVM-wide defaults** that affect ALL `HttpsURLConnection` instances in the entire process — not just the FE-to-FE connection being created. This is problematic because:
1. **Thread safety**: Multiple threads calling this concurrently could race. There's no synchronization.
2. **Global hostname verification bypass**: `(hostname, session) -> true` disables hostname verification for ALL future `HttpsURLConnection` instances, including any external catalog connections or other HTTPS clients using `HttpsURLConnection`.
3. **Redundant re-initialization**: This is called on every `getConnectionWithNodeIdent()` invocation, re-reading the truststore from disk and re-initializing the `SSLContext` each time.
Recommendation: Initialize the `SSLContext` and install the defaults **once** at startup (e.g., via a static initializer or an `init()` method called during FE bootstrap). Alternatively, set the SSL factory on the individual `HttpsURLConnection` instance rather than the global default:
```java
HttpsURLConnection httpsConn = (HttpsURLConnection) url.openConnection();
httpsConn.setSSLSocketFactory(sslContext.getSocketFactory());
httpsConn.setHostnameVerifier(...);
```
This avoids polluting the JVM-wide defaults.
##########
fe/fe-core/src/main/java/org/apache/doris/master/MetaHelper.java:
##########
@@ -39,7 +39,6 @@
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.HttpURLConnection;
-import java.util.Map;
public class MetaHelper {
public static final Logger LOG = LogManager.getLogger(MetaHelper.class);
Review Comment:
**Removed useful debug logging**
The `LOG.info("meta helper, url: {}, timeout{}, headers: {}", url, timeout, headers)` was removed. While the response wrapping in try-catch is a good improvement, the debug logging for URL, timeout, and headers is valuable for troubleshooting connectivity issues (especially with this HTTPS change). Consider keeping it, perhaps at `LOG.debug` level.
##########
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/manager/HttpUtils.java:
##########
@@ -130,7 +131,14 @@ public static CloseableHttpClient getHttpClient() {
}
private static String executeRequest(HttpRequestBase request) throws IOException {
- CloseableHttpClient client = getHttpClient();
+ CloseableHttpClient client;
Review Comment:
**Bug: Resource leak — `CloseableHttpClient` is never closed**
The `CloseableHttpClient` created here (both the HTTPS and plain HTTP paths) is never closed. The `ResponseHandler` variant of `execute()` closes the HTTP *response*, but NOT the *client* itself. This leaks the client's internal `PoolingHttpClientConnectionManager` and associated socket file descriptors.
Other callers in the codebase do this correctly (e.g., `MysqlLoadManager.java` uses `try-with-resources`).
Fix:
```java
private static String executeRequest(HttpRequestBase request) throws IOException {
try (CloseableHttpClient client = request.getURI().getScheme().equalsIgnoreCase("https")
&& Config.enable_https
? InternalHttpsUtils.createValidatedHttpClient()
: HttpClientBuilder.create().build()) {
return client.execute(request, httpResponse -> EntityUtils.toString(httpResponse.getEntity()));
}
}
```
Note: this resource leak also existed before this PR (the old code called `getHttpClient()` without closing it), but since this PR rewrites the method, it's a good opportunity to fix it.
##########
fe/fe-core/src/main/java/org/apache/doris/common/util/InternalHttpsUtils.java:
##########
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.util;
+
+import org.apache.doris.common.Config;
+
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.security.KeyStore;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+
+/**
+ * SSL-aware HTTP clients for internal FE communication using MySQL SSL truststore.
+ *
+ * Security Model:
+ * - Validates certificates against configured CA truststore (mysql_ssl_default_ca_certificate)
+ * - Hostname verification is DISABLED to support IP-based FE communication
+ * - This is safe for internal cluster communication because:
+ * 1. All endpoints enforce checkFromValidFe() - only registered FE nodes can connect
+ * 2. FE cluster is assumed to be on trusted network
+ * 3. Traffic is encrypted and authenticated via certificate validation
+ *
+ * This approach is similar to other distributed systems (Kafka, Elasticsearch, Cassandra)
+ * where inter-node SSL communication disables hostname verification for operational flexibility.
+ */
+public class InternalHttpsUtils {
+ private static final Logger LOG = LogManager.getLogger(InternalHttpsUtils.class);
+
+ public static CloseableHttpClient createValidatedHttpClient() {
+ try {
Review Comment:
**Performance: Truststore re-read on every call**
Both `createValidatedHttpClient()` and `installTrustManagerForUrlConnection()` re-read the truststore file from disk and re-initialize the `SSLContext` on every invocation. For internal FE-to-FE calls (checkpoint sync, journal ID queries, etc.), this could be called frequently.
Consider caching the `SSLContext` in a static field, initialized lazily or at startup. The truststore file doesn't change at runtime.
##########
fe/fe-core/src/main/java/org/apache/doris/common/util/HttpURLUtil.java:
##########
@@ -34,9 +35,12 @@ public static HttpURLConnection getConnectionWithNodeIdent(String request) throw
try {
SecurityChecker.getInstance().startSSRFChecking(request);
URL url = new URL(request);
+
+ if (url.getProtocol().equalsIgnoreCase("https") && Config.enable_https) {
+ InternalHttpsUtils.installTrustManagerForUrlConnection();
+ }
+
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
- // Must use Env.getServingEnv() instead of getCurrentEnv(),
- // because here we need to obtain selfNode through the official service catalog.
HostInfo selfNode = Env.getServingEnv().getSelfNode();
conn.setRequestProperty(Env.CLIENT_NODE_HOST_KEY, selfNode.getHost());
conn.setRequestProperty(Env.CLIENT_NODE_PORT_KEY, selfNode.getPort() + "");
Review Comment:
**Deleted useful comments**
The comments `// Must use Env.getServingEnv() instead of getCurrentEnv(), // because here we need to obtain selfNode through the official service catalog.` were removed here and in `getNodeIdentHeaders()`. These documented a non-obvious architectural requirement. Please preserve them — they explain *why* `getServingEnv()` is used instead of the more common `getCurrentEnv()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r3364371392
##########
fe/fe-core/src/main/java/org/apache/doris/master/MetaHelper.java:
##########
@@ -149,9 +149,13 @@ private static void checkFile(File file) throws IOException {
public static <T> ResponseBody doGet(String url, int timeout, Class<T> clazz) throws IOException {
Map<String, String> headers = HttpURLUtil.getNodeIdentHeaders();
- LOG.info("meta helper, url: {}, timeout{}, headers: {}", url, timeout, headers);
+ LOG.debug("meta helper, url: {}, timeout: {}, headers: {}", url, timeout, headers);
String response = HttpUtils.doGet(url, headers, timeout);
Review Comment:
Thanks for suggestion, reverted to use info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4673468664
@morrySnow please could you help with a review and merging this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4586368735
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4611180447
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4587212996
<details>
<summary>TPC-H: <b>Total hot run time: 31224 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5c1f94c0bd3128aced58ce172868708101d080d7, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17656 4107 4042 4042
q2 q3 10744 1424 835 835
q4 4684 475 352 352
q5 7574 2222 2114 2114
q6 275 181 141 141
q7 956 779 642 642
q8 9369 1689 1541 1541
q9 6700 5009 4948 4948
q10 6430 2241 1853 1853
q11 427 271 246 246
q12 695 431 295 295
q13 18185 3404 2766 2766
q14 268 259 237 237
q15 q16 822 774 715 715
q17 993 937 949 937
q18 6799 5776 5537 5537
q19 1271 1271 1111 1111
q20 529 403 261 261
q21 5660 2601 2347 2347
q22 454 357 304 304
Total cold run time: 100491 ms
Total hot run time: 31224 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4448 4341 4393 4341
q2 q3 4559 4925 4339 4339
q4 2101 2220 1384 1384
q5 4481 4321 4777 4321
q6 255 195 159 159
q7 2083 1922 1612 1612
q8 2473 2174 2304 2174
q9 7849 8118 7869 7869
q10 4863 4805 4388 4388
q11 590 459 408 408
q12 750 742 539 539
q13 3342 3638 2996 2996
q14 307 293 281 281
q15 q16 748 768 664 664
q17 1388 1398 1365 1365
q18 7835 7439 6822 6822
q19 1100 1078 1110 1078
q20 2214 2217 1964 1964
q21 5321 4607 4540 4540
q22 544 462 437 437
Total cold run time: 57251 ms
Total hot run time: 51681 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4719535456
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r2882558584
##########
fe/fe-core/src/main/java/org/apache/doris/common/util/HttpURLUtil.java:
##########
@@ -34,9 +35,12 @@ public static HttpURLConnection getConnectionWithNodeIdent(String request) throw
try {
SecurityChecker.getInstance().startSSRFChecking(request);
URL url = new URL(request);
+
+ if (url.getProtocol().equalsIgnoreCase("https") && Config.enable_https) {
+ InternalHttpsUtils.installTrustManagerForUrlConnection();
+ }
+
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
- // Must use Env.getServingEnv() instead of getCurrentEnv(),
- // because here we need to obtain selfNode through the official service catalog.
HostInfo selfNode = Env.getServingEnv().getSelfNode();
conn.setRequestProperty(Env.CLIENT_NODE_HOST_KEY, selfNode.getHost());
conn.setRequestProperty(Env.CLIENT_NODE_PORT_KEY, selfNode.getPort() + "");
Review Comment:
Restored comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-3996178501
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4611179984
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4725259710
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4726585267
@morningman Thanks for the review. Could you please trigger /review one more time? I've addressed all identified scenarios, but I'd appreciate another pass to catch any edge cases. If everything looks good, please help review and merge the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4633096961
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4657842735
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4671794816
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-3988580364
<details>
<summary>TPC-DS: <b>Total hot run time: 184255 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 29f45dbe504c8d46be0de681c14f24d3a97c8291, data reload: false
query5 4327 636 512 512
query6 321 222 211 211
query7 4233 489 282 282
query8 359 255 235 235
query9 9079 2842 2836 2836
query10 512 428 363 363
query11 17107 17503 17564 17503
query12 201 132 129 129
query13 1269 491 360 360
query14 7404 3267 3012 3012
query14_1 2935 2969 2900 2900
query15 216 192 190 190
query16 1026 407 478 407
query17 1179 782 661 661
query18 2873 495 375 375
query19 215 229 191 191
query20 144 132 136 132
query21 217 136 121 121
query22 5997 5318 4970 4970
query23 17227 16838 16732 16732
query23_1 16798 16803 16807 16803
query24 7063 1635 1224 1224
query24_1 1249 1252 1233 1233
query25 547 481 428 428
query26 821 281 160 160
query27 2719 484 280 280
query28 4377 1860 1856 1856
query29 783 562 471 471
query30 311 235 208 208
query31 855 754 618 618
query32 81 71 68 68
query33 516 325 279 279
query34 883 915 572 572
query35 623 672 586 586
query36 1090 1148 972 972
query37 133 93 82 82
query38 3000 2907 2877 2877
query39 899 871 836 836
query39_1 825 836 832 832
query40 223 149 135 135
query41 66 59 58 58
query42 104 104 101 101
query43 369 391 353 353
query44
query45 198 192 182 182
query46 924 986 603 603
query47 2161 2183 2067 2067
query48 317 312 236 236
query49 615 481 372 372
query50 679 276 215 215
query51 4076 4133 4008 4008
query52 108 108 97 97
query53 290 338 300 300
query54 282 267 259 259
query55 88 83 80 80
query56 343 287 305 287
query57 1359 1371 1283 1283
query58 288 277 284 277
query59 2608 2682 2503 2503
query60 335 330 314 314
query61 147 145 144 144
query62 641 591 550 550
query63 307 277 284 277
query64 4163 1255 1002 1002
query65
query66 1352 450 358 358
query67 16237 16225 16247 16225
query68
query69 393 312 281 281
query70 1006 962 963 962
query71 335 318 285 285
query72 2771 2678 2430 2430
query73 527 551 317 317
query74 10015 9921 9765 9765
query75 2854 2733 2455 2455
query76 2322 1029 696 696
query77 355 377 308 308
query78 11162 11329 10664 10664
query79 2678 791 605 605
query80 1780 621 576 576
query81 567 270 244 244
query82 1015 148 119 119
query83 348 260 243 243
query84 251 114 101 101
query85 870 485 439 439
query86 415 300 295 295
query87 3104 3101 2988 2988
query88 3510 2663 2656 2656
query89 427 368 344 344
query90 1951 167 168 167
query91 162 153 141 141
query92 74 77 69 69
query93 1090 823 507 507
query94 628 321 296 296
query95 594 395 315 315
query96 643 505 229 229
query97 2462 2480 2455 2455
query98 227 233 218 218
query99 1048 947 916 916
Total cold run time: 255584 ms
Total hot run time: 184255 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4072459703
@morningman @CalvinKirs @dataroaring : Could you please review and approve !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r2882562318
##########
fe/fe-core/src/main/java/org/apache/doris/common/util/InternalHttpsUtils.java:
##########
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.util;
+
+import org.apache.doris.common.Config;
+
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.security.KeyStore;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+
+/**
+ * SSL-aware HTTP clients for internal FE communication using MySQL SSL truststore.
+ *
+ * Security Model:
+ * - Validates certificates against configured CA truststore (mysql_ssl_default_ca_certificate)
+ * - Hostname verification is DISABLED to support IP-based FE communication
+ * - This is safe for internal cluster communication because:
+ * 1. All endpoints enforce checkFromValidFe() - only registered FE nodes can connect
+ * 2. FE cluster is assumed to be on trusted network
+ * 3. Traffic is encrypted and authenticated via certificate validation
+ *
+ * This approach is similar to other distributed systems (Kafka, Elasticsearch, Cassandra)
+ * where inter-node SSL communication disables hostname verification for operational flexibility.
+ */
+public class InternalHttpsUtils {
+ private static final Logger LOG = LogManager.getLogger(InternalHttpsUtils.class);
+
+ public static CloseableHttpClient createValidatedHttpClient() {
+ try {
Review Comment:
Handled to read once and cache, refresh is always done via FE service restart to fetch at startup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4028455105
@Hastyshell @dataroaring @morningman @CalvinKirs - Could you please help with review and get this merged !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4587124371
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4586791118
<details>
<summary>TPC-DS: <b>Total hot run time: 172375 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 749eb56236b404633b3b5fc012b5f3e55f7de570, data reload: false
query5 4325 669 518 518
query6 336 214 201 201
query7 4227 584 306 306
query8 324 232 222 222
query9 8785 4027 3986 3986
query10 466 347 297 297
query11 5773 2562 2209 2209
query12 204 127 127 127
query13 1285 611 436 436
query14 6044 5533 5137 5137
query14_1 4482 4438 4449 4438
query15 211 208 185 185
query16 981 460 420 420
query17 949 707 606 606
query18 2439 488 356 356
query19 212 200 167 167
query20 136 135 132 132
query21 215 146 116 116
query22 13737 13695 13445 13445
query23 17323 16576 16129 16129
query23_1 16464 16439 16367 16367
query24 7541 1812 1357 1357
query24_1 1310 1338 1357 1338
query25 571 500 442 442
query26 1329 326 185 185
query27 2733 557 354 354
query28 4439 2012 1998 1998
query29 1004 651 523 523
query30 327 248 204 204
query31 1113 1085 958 958
query32 91 81 78 78
query33 564 406 309 309
query34 1185 1148 649 649
query35 786 813 712 712
query36 1396 1437 1229 1229
query37 156 107 96 96
query38 3209 3135 3089 3089
query39 924 915 896 896
query39_1 894 902 898 898
query40 234 155 134 134
query41 72 71 69 69
query42 114 113 109 109
query43 335 339 308 308
query44
query45 219 208 198 198
query46 1104 1203 759 759
query47 2346 2386 2233 2233
query48 418 446 328 328
query49 666 514 419 419
query50 976 349 256 256
query51 4360 4332 4269 4269
query52 114 109 95 95
query53 261 292 213 213
query54 334 286 286 286
query55 96 96 90 90
query56 311 330 330 330
query57 1472 1447 1341 1341
query58 319 290 284 284
query59 1595 1705 1442 1442
query60 311 316 312 312
query61 159 158 156 156
query62 706 648 592 592
query63 243 208 210 208
query64 2406 804 641 641
query65
query66 1719 488 361 361
query67 29811 29837 29641 29641
query68
query69 481 354 306 306
query70 979 992 980 980
query71 309 274 261 261
query72 2981 2735 2390 2390
query73 837 813 440 440
query74 5135 4944 4808 4808
query75 2708 2610 2278 2278
query76 2268 1171 791 791
query77 399 425 337 337
query78 12422 12492 11752 11752
query79 1492 1057 759 759
query80 660 538 476 476
query81 459 283 239 239
query82 1401 159 122 122
query83 378 283 255 255
query84 311 140 115 115
query85 931 547 475 475
query86 415 331 368 331
query87 3457 3383 3228 3228
query88 3601 2727 2719 2719
query89 447 395 352 352
query90 1978 189 185 185
query91 183 171 145 145
query92 81 83 75 75
query93 1471 1543 910 910
query94 539 371 317 317
query95 664 403 351 351
query96 1070 790 327 327
query97 2729 2724 2595 2595
query98 253 228 228 228
query99 1187 1172 1044 1044
Total cold run time: 254358 ms
Total hot run time: 172375 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r3422321418
##########
fe/fe-core/src/main/java/org/apache/doris/common/util/InternalHttpsUtils.java:
##########
@@ -0,0 +1,98 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.util;
+
+import org.apache.doris.common.Config;
+
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.security.KeyStore;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+
+/**
+ * Utility for creating SSL-aware HTTP clients for internal FE-to-FE communication.
+ *
+ * <p>Builds an {@link SSLContext} from the configured CA truststore once and caches it.
+ * Hostname verification is disabled for IP-based intra-cluster connections.
+ * Certificate rotation requires a FE restart.
+ */
+public class InternalHttpsUtils {
+ private static volatile SSLContext cachedSslContext = null;
+
+ private static final Logger LOG = LogManager.getLogger(InternalHttpsUtils.class);
+
+ /**
+ * Returns the cached SSLContext, building it from the configured truststore on first call.
+ */
+ public static SSLContext getSslContext() {
+ if (cachedSslContext == null) {
+ synchronized (InternalHttpsUtils.class) {
+ if (cachedSslContext == null) {
+ cachedSslContext = buildSslContext();
+ }
+ }
+ }
+ return cachedSslContext;
+ }
+
+ private static SSLContext buildSslContext() {
+ try {
+ KeyStore trustStore = KeyStore.getInstance(Config.ssl_trust_store_type);
+ try (InputStream stream = Files.newInputStream(
+ Paths.get(Config.mysql_ssl_default_ca_certificate))) {
+ trustStore.load(stream, Config.mysql_ssl_default_ca_certificate_password.toCharArray());
Review Comment:
In a standard Doris SSL setup, a single internal CA is generated locally and used to sign all TLS endpoints. In enterprise deployments, an enterprise PKI CA serves the same role — one CA issues all certificates across MySQL SSL, FE HTTPS, and other endpoints. mysql_ssl_default_ca_certificate holds that CA cert, making it the correct trust anchor for validating FE-to-FE HTTPS connections in both cases. The current code is intentional.
I'll add a comment to InternalHttpsUtils.buildSslContext() to make this assumption explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4633310418
<details>
<summary>TPC-H: <b>Total hot run time: 28979 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 052c6fd279371dfb63b1b5d048c7f082653ef511, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17617 3982 3951 3951
q2 q3 10748 1345 810 810
q4 4685 475 339 339
q5 7602 882 601 601
q6 181 168 136 136
q7 796 845 645 645
q8 9567 1519 1624 1519
q9 6888 4468 4448 4448
q10 6826 1810 1534 1534
q11 428 266 243 243
q12 675 418 288 288
q13 18158 3404 2786 2786
q14 280 261 253 253
q15 q16 825 773 702 702
q17 1342 1124 830 830
q18 6734 5748 5607 5607
q19 1355 1268 1042 1042
q20 546 398 264 264
q21 5983 2855 2663 2663
q22 450 368 318 318
Total cold run time: 101686 ms
Total hot run time: 28979 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4853 4705 4640 4640
q2 q3 4994 5314 4637 4637
q4 2099 2189 1413 1413
q5 4852 4725 4819 4725
q6 233 182 133 133
q7 1855 1731 1559 1559
q8 2606 2087 1932 1932
q9 7371 7377 7631 7377
q10 4714 4670 4199 4199
q11 537 379 354 354
q12 731 747 522 522
q13 2979 3383 2807 2807
q14 270 293 267 267
q15 q16 677 694 602 602
q17 1277 1250 1238 1238
q18 7348 6745 6908 6745
q19 1107 1164 1099 1099
q20 2212 2226 1942 1942
q21 5253 5045 4423 4423
q22 509 487 402 402
Total cold run time: 56477 ms
Total hot run time: 51016 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r3389384074
##########
fe/fe-core/src/main/java/org/apache/doris/system/HeartbeatMgr.java:
##########
@@ -96,7 +97,7 @@ public void setMaster(int clusterId, String token, long epoch) {
TMasterInfo tMasterInfo = new TMasterInfo(
new TNetworkAddress(FrontendOptions.getLocalHostAddress(), Config.rpc_port), clusterId, epoch);
tMasterInfo.setToken(token);
- tMasterInfo.setHttpPort(Config.http_port);
+ tMasterInfo.setHttpPort(HttpURLUtil.getHttpPort());
long flags = heartbeatFlags.getHeartbeatFlags();
tMasterInfo.setHeartbeatFlags(flags);
Review Comment:
This PR and #63918 are intended to be consumed together; the BE-side handling required for HTTPS small-file downloads is already in place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r2882553339
##########
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/manager/HttpUtils.java:
##########
@@ -130,7 +131,14 @@ public static CloseableHttpClient getHttpClient() {
}
private static String executeRequest(HttpRequestBase request) throws IOException {
- CloseableHttpClient client = getHttpClient();
+ CloseableHttpClient client;
Review Comment:
Good suggestion, Implemented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4671991843
<details>
<summary>TPC-H: <b>Total hot run time: 29359 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f798ef8d547cec261161ace7a3c6508994d20787, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17667 4178 3982 3982
q2 q3 10744 1401 798 798
q4 4683 473 339 339
q5 7518 869 589 589
q6 183 174 137 137
q7 760 842 648 648
q8 9859 1618 1628 1618
q9 6650 4451 4427 4427
q10 6749 1791 1506 1506
q11 434 263 252 252
q12 630 430 302 302
q13 18112 3754 2844 2844
q14 274 258 248 248
q15 q16 829 769 704 704
q17 997 935 917 917
q18 6780 5821 5699 5699
q19 1216 1261 1072 1072
q20 512 394 253 253
q21 5934 2850 2696 2696
q22 461 386 328 328
Total cold run time: 100992 ms
Total hot run time: 29359 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4914 4756 4755 4755
q2 q3 5134 5195 4618 4618
q4 2113 2175 1374 1374
q5 4906 4621 4826 4621
q6 228 178 132 132
q7 1866 1713 1607 1607
q8 2480 2116 1988 1988
q9 7471 7463 7392 7392
q10 4729 4653 4190 4190
q11 546 405 363 363
q12 726 749 533 533
q13 3013 3360 2824 2824
q14 275 281 250 250
q15 q16 674 704 619 619
q17 1286 1251 1273 1251
q18 7399 6898 6787 6787
q19 1106 1089 1077 1077
q20 2241 2215 1948 1948
q21 5296 4610 4460 4460
q22 538 474 409 409
Total cold run time: 56941 ms
Total hot run time: 51198 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4672063315
<details>
<summary>TPC-DS: <b>Total hot run time: 169035 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f798ef8d547cec261161ace7a3c6508994d20787, data reload: false
query5 4348 639 484 484
query6 457 204 180 180
query7 4823 547 299 299
query8 361 212 196 196
query9 8774 4251 4170 4170
query10 458 326 265 265
query11 5954 2383 2117 2117
query12 164 105 101 101
query13 1279 572 441 441
query14 6419 5385 5062 5062
query14_1 4392 4375 4389 4375
query15 209 200 180 180
query16 1050 460 359 359
query17 1153 723 593 593
query18 2673 473 345 345
query19 232 187 162 162
query20 125 110 106 106
query21 232 138 121 121
query22 13621 13622 13293 13293
query23 17326 16554 16088 16088
query23_1 16264 16259 16339 16259
query24 7681 1724 1325 1325
query24_1 1330 1306 1326 1306
query25 578 469 405 405
query26 1326 321 175 175
query27 2641 522 325 325
query28 4418 2102 2055 2055
query29 1139 628 501 501
query30 319 244 205 205
query31 1120 1086 964 964
query32 115 67 61 61
query33 542 335 274 274
query34 1224 1152 666 666
query35 770 808 668 668
query36 1415 1382 1249 1249
query37 156 101 88 88
query38 3203 3146 3042 3042
query39 926 924 898 898
query39_1 881 868 873 868
query40 229 121 96 96
query41 63 62 60 60
query42 94 92 93 92
query43 331 320 285 285
query44
query45 194 186 178 178
query46 1072 1169 729 729
query47 2358 2358 2275 2275
query48 367 425 291 291
query49 627 470 351 351
query50 990 345 243 243
query51 4361 4337 4229 4229
query52 87 86 75 75
query53 254 264 193 193
query54 272 217 192 192
query55 78 75 72 72
query56 236 216 224 216
query57 1411 1393 1308 1308
query58 239 212 205 205
query59 1570 1658 1430 1430
query60 278 242 229 229
query61 159 156 149 149
query62 697 652 594 594
query63 227 186 183 183
query64 2535 788 623 623
query65
query66 1783 457 347 347
query67 29582 29593 29406 29406
query68
query69 425 308 260 260
query70 1035 906 970 906
query71 292 225 210 210
query72 2899 2680 2371 2371
query73 848 777 451 451
query74 5122 4982 4736 4736
query75 2617 2558 2244 2244
query76 2332 1140 800 800
query77 365 363 280 280
query78 12282 12322 11871 11871
query79 1557 1069 788 788
query80 1271 488 378 378
query81 546 277 243 243
query82 608 155 115 115
query83 328 277 242 242
query84
query85 969 533 438 438
query86 459 308 279 279
query87 3393 3406 3184 3184
query88 3655 2797 2779 2779
query89 426 390 343 343
query90 1955 179 176 176
query91 180 166 136 136
query92 65 61 59 59
query93 1623 1378 941 941
query94 741 366 321 321
query95 687 488 352 352
query96 1015 798 341 341
query97 2698 2725 2551 2551
query98 208 206 200 200
query99 1167 1177 1032 1032
Total cold run time: 252528 ms
Total hot run time: 169035 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4725935349
<details>
<summary>ClickBench: <b>Total hot run time: 25.2 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit affe1d2a5247d28e273bbc3132b34306e9142694, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.26 0.14 0.12
query4 1.61 0.12 0.14
query5 0.24 0.22 0.21
query6 1.23 1.08 1.10
query7 0.04 0.00 0.00
query8 0.07 0.04 0.04
query9 0.38 0.35 0.31
query10 0.54 0.54 0.53
query11 0.20 0.14 0.13
query12 0.18 0.16 0.15
query13 0.47 0.47 0.48
query14 1.01 1.01 1.02
query15 0.62 0.60 0.60
query16 0.32 0.32 0.34
query17 1.11 1.10 1.11
query18 0.23 0.21 0.21
query19 1.99 1.96 1.91
query20 0.02 0.01 0.01
query21 15.43 0.21 0.12
query22 4.95 0.06 0.05
query23 16.10 0.31 0.12
query24 2.92 0.42 0.33
query25 0.11 0.04 0.05
query26 0.72 0.21 0.15
query27 0.04 0.04 0.04
query28 3.49 0.88 0.52
query29 12.49 4.37 3.51
query30 0.27 0.15 0.15
query31 2.78 0.59 0.31
query32 3.23 0.59 0.49
query33 3.22 3.19 3.31
query34 15.56 4.23 3.50
query35 3.54 3.52 3.50
query36 0.57 0.42 0.44
query37 0.10 0.07 0.07
query38 0.06 0.04 0.04
query39 0.05 0.03 0.03
query40 0.20 0.16 0.15
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 96.62 s
Total hot run time: 25.2 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4725831584
<details>
<summary>TPC-H: <b>Total hot run time: 29036 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit affe1d2a5247d28e273bbc3132b34306e9142694, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17964 4096 4047 4047
q2 2049 303 179 179
q3 10306 1360 815 815
q4 4683 467 338 338
q5 7477 833 565 565
q6 191 168 133 133
q7 761 833 614 614
q8 9733 1606 1573 1573
q9 6175 4457 4440 4440
q10 6847 1775 1518 1518
q11 432 265 233 233
q12 639 411 302 302
q13 18205 3337 2751 2751
q14 259 252 252 252
q15 q16 777 765 711 711
q17 933 876 862 862
q18 6844 5702 5625 5625
q19 1309 1385 1083 1083
q20 479 397 260 260
q21 5852 2631 2439 2439
q22 424 367 296 296
Total cold run time: 102339 ms
Total hot run time: 29036 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4405 4271 4345 4271
q2 327 346 233 233
q3 4562 4918 4410 4410
q4 2052 2157 1358 1358
q5 4502 4345 4352 4345
q6 229 172 125 125
q7 1746 1750 1651 1651
q8 2548 2243 2123 2123
q9 8002 8102 7846 7846
q10 4801 4750 4300 4300
q11 619 428 395 395
q12 744 757 539 539
q13 3520 3561 3055 3055
q14 309 316 265 265
q15 q16 709 771 655 655
q17 1396 1380 1359 1359
q18 8035 7383 7009 7009
q19 1108 1098 1070 1070
q20 2204 2224 1945 1945
q21 5251 4526 4400 4400
q22 507 459 398 398
Total cold run time: 57576 ms
Total hot run time: 51752 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #60921:
URL: https://github.com/apache/doris/pull/60921
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4587615044
# FE Regression Coverage Report
Increment line coverage ` 17.07% (14/82)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60921_5c1f94c0bd3128aced58ce172868708101d080d7_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60921_5c1f94c0bd3128aced58ce172868708101d080d7_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4586820696
# FE UT Coverage Report
Increment line coverage ` 21.95% (18/82)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60921_749eb56236b404633b3b5fc012b5f3e55f7de570/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60921_749eb56236b404633b3b5fc012b5f3e55f7de570/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4603752955
<details>
<summary>TPC-DS: <b>Total hot run time: 169471 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9172b9e209226839f6eaa2002c4d250013140767, data reload: false
query5 4356 621 497 497
query6 442 202 193 193
query7 4868 586 302 302
query8 371 216 210 210
query9 8805 4089 4079 4079
query10 457 315 268 268
query11 5929 2365 2216 2216
query12 160 104 100 100
query13 1293 576 444 444
query14 6397 5411 5082 5082
query14_1 4415 4401 4413 4401
query15 210 202 179 179
query16 993 449 479 449
query17 1108 695 563 563
query18 2522 468 344 344
query19 198 181 139 139
query20 113 107 104 104
query21 209 136 116 116
query22 13673 13531 13374 13374
query23 17417 16747 16186 16186
query23_1 16193 16194 16295 16194
query24 7433 1762 1314 1314
query24_1 1297 1293 1309 1293
query25 546 440 387 387
query26 1306 303 173 173
query27 2678 570 328 328
query28 4449 2012 2016 2012
query29 1092 618 488 488
query30 308 235 203 203
query31 1125 1066 955 955
query32 109 59 59 59
query33 499 317 253 253
query34 1187 1181 648 648
query35 761 773 676 676
query36 1404 1408 1215 1215
query37 156 104 89 89
query38 3223 3118 3063 3063
query39 921 933 902 902
query39_1 890 861 879 861
query40 229 129 107 107
query41 71 70 70 70
query42 97 96 95 95
query43 321 324 282 282
query44
query45 203 189 186 186
query46 1102 1190 738 738
query47 2343 2402 2233 2233
query48 423 421 313 313
query49 674 485 374 374
query50 1030 358 259 259
query51 4402 4300 4273 4273
query52 93 92 78 78
query53 247 267 196 196
query54 278 234 227 227
query55 82 79 73 73
query56 245 276 241 241
query57 1447 1399 1335 1335
query58 256 226 225 225
query59 1612 1678 1425 1425
query60 286 269 245 245
query61 186 185 184 184
query62 716 648 577 577
query63 233 186 186 186
query64 2604 842 680 680
query65
query66 1813 464 342 342
query67 29795 29700 29581 29581
query68
query69 426 304 268 268
query70 1002 969 949 949
query71 317 221 215 215
query72 3040 2679 2364 2364
query73 857 783 460 460
query74 5123 4980 4766 4766
query75 2654 2599 2251 2251
query76 2304 1162 765 765
query77 349 368 281 281
query78 12357 12383 11938 11938
query79 1342 1075 794 794
query80 591 475 404 404
query81 459 281 239 239
query82 562 158 118 118
query83 352 282 252 252
query84 253 143 116 116
query85 905 553 440 440
query86 371 301 291 291
query87 3381 3361 3182 3182
query88 3661 2757 2717 2717
query89 432 385 330 330
query90 1925 186 193 186
query91 180 176 135 135
query92 64 62 59 59
query93 1487 1424 943 943
query94 528 353 314 314
query95 706 375 438 375
query96 1046 845 351 351
query97 2705 2719 2545 2545
query98 213 205 202 202
query99 1178 1191 1022 1022
Total cold run time: 251533 ms
Total hot run time: 169471 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-3977606404
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r3340277953
##########
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/manager/QueryProfileAction.java:
##########
@@ -196,7 +196,7 @@ public Object queryInfo(HttpServletRequest request, HttpServletResponse response
// add node information
for (List<String> query : queries) {
query.set(1, NetUtils.getHostPortInAccessibleFormat(Env.getCurrentEnv().getSelfNode().getHost(),
- Config.http_port));
+ Config.enable_https ? Config.https_port : Config.http_port));
Review Comment:
Thanks @morningman , Good suggestion. I didn't realise such many ternary used. Now handled with HttpURLUtil.getHttpPort() method as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4672399984
# FE UT Coverage Report
Increment line coverage ` 23.81% (20/84)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60921_f798ef8d547cec261161ace7a3c6508994d20787/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60921_f798ef8d547cec261161ace7a3c6508994d20787/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "Hastyshell (via GitHub)" <gi...@apache.org>.
Hastyshell commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-3989555197
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r2882557138
##########
fe/fe-core/src/main/java/org/apache/doris/common/util/InternalHttpsUtils.java:
##########
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.util;
+
+import org.apache.doris.common.Config;
+
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.security.KeyStore;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.TrustManagerFactory;
+
+/**
+ * SSL-aware HTTP clients for internal FE communication using MySQL SSL truststore.
+ *
+ * Security Model:
+ * - Validates certificates against configured CA truststore (mysql_ssl_default_ca_certificate)
+ * - Hostname verification is DISABLED to support IP-based FE communication
+ * - This is safe for internal cluster communication because:
+ * 1. All endpoints enforce checkFromValidFe() - only registered FE nodes can connect
+ * 2. FE cluster is assumed to be on trusted network
+ * 3. Traffic is encrypted and authenticated via certificate validation
+ *
+ * This approach is similar to other distributed systems (Kafka, Elasticsearch, Cassandra)
+ * where inter-node SSL communication disables hostname verification for operational flexibility.
+ */
+public class InternalHttpsUtils {
+ private static final Logger LOG = LogManager.getLogger(InternalHttpsUtils.class);
+
+ public static CloseableHttpClient createValidatedHttpClient() {
+ try {
+ KeyStore trustStore = KeyStore.getInstance(Config.ssl_trust_store_type);
+ try (InputStream stream = Files.newInputStream(
+ Paths.get(Config.mysql_ssl_default_ca_certificate))) {
+ trustStore.load(stream, Config.mysql_ssl_default_ca_certificate_password.toCharArray());
+ }
+
+ TrustManagerFactory tmf = TrustManagerFactory.getInstance(
+ TrustManagerFactory.getDefaultAlgorithm());
+ tmf.init(trustStore);
+
+ SSLContext sslContext = SSLContext.getInstance("TLS");
+ sslContext.init(null, tmf.getTrustManagers(), null);
+
+ SSLConnectionSocketFactory sslFactory = new SSLConnectionSocketFactory(
+ sslContext,
+ NoopHostnameVerifier.INSTANCE);
+
+ return HttpClients.custom()
+ .setSSLSocketFactory(sslFactory)
+ .build();
+ } catch (Exception e) {
+ LOG.error("Failed to create SSL-aware HTTP client using truststore: {}",
+ Config.mysql_ssl_default_ca_certificate, e);
+ throw new RuntimeException("Failed to create SSL-aware HTTP client", e);
+ }
+ }
+
+ public static void installTrustManagerForUrlConnection() {
+ try {
+ KeyStore trustStore = KeyStore.getInstance(Config.ssl_trust_store_type);
+ try (InputStream stream = Files.newInputStream(
+ Paths.get(Config.mysql_ssl_default_ca_certificate))) {
+ trustStore.load(stream, Config.mysql_ssl_default_ca_certificate_password.toCharArray());
+ }
+
+ TrustManagerFactory tmf = TrustManagerFactory.getInstance(
+ TrustManagerFactory.getDefaultAlgorithm());
+ tmf.init(trustStore);
+
+ SSLContext sslContext = SSLContext.getInstance("TLS");
+ sslContext.init(null, tmf.getTrustManagers(), null);
+
+ javax.net.ssl.HttpsURLConnection.setDefaultSSLSocketFactory(sslContext.getSocketFactory());
+ javax.net.ssl.HttpsURLConnection.setDefaultHostnameVerifier((hostname, session) -> true);
Review Comment:
Suggestion implemented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4842533096
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4603653237
<details>
<summary>TPC-H: <b>Total hot run time: 29054 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9172b9e209226839f6eaa2002c4d250013140767, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17666 4018 3986 3986
q2 q3 10820 1367 788 788
q4 4689 476 352 352
q5 7603 916 592 592
q6 180 169 134 134
q7 757 869 655 655
q8 9358 1595 1636 1595
q9 5767 4477 4487 4477
q10 6746 1819 1527 1527
q11 433 264 241 241
q12 624 421 283 283
q13 18237 3356 2772 2772
q14 261 260 246 246
q15 q16 808 771 716 716
q17 985 1021 976 976
q18 6732 5628 5516 5516
q19 1202 1326 973 973
q20 514 401 266 266
q21 6107 2753 2640 2640
q22 475 385 319 319
Total cold run time: 99964 ms
Total hot run time: 29054 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5092 4693 4832 4693
q2 q3 4843 5305 4670 4670
q4 2092 2185 1364 1364
q5 4751 4910 4718 4718
q6 226 175 131 131
q7 1842 1791 1562 1562
q8 2410 2080 2065 2065
q9 7964 7579 7392 7392
q10 4763 4713 4194 4194
q11 524 379 351 351
q12 728 734 520 520
q13 3016 3337 2815 2815
q14 280 287 247 247
q15 q16 670 697 600 600
q17 1266 1241 1249 1241
q18 7397 6829 6872 6829
q19 1130 1077 1095 1077
q20 2234 2213 1925 1925
q21 5228 4512 4362 4362
q22 525 487 408 408
Total cold run time: 56981 ms
Total hot run time: 51164 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4601971699
# FE UT Coverage Report
Increment line coverage ` 24.10% (20/83)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60921_9172b9e209226839f6eaa2002c4d250013140767/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60921_9172b9e209226839f6eaa2002c4d250013140767/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4637184581
run nonconcurrent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4637524903
run nonConcurrent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-3988444088
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4009250686
@Hastyshell @dataroaring @morningman @CalvinKirs - Could you please help with review and get this merged !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4587104366
# FE Regression Coverage Report
Increment line coverage ` 17.07% (14/82)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60921_749eb56236b404633b3b5fc012b5f3e55f7de570_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60921_749eb56236b404633b3b5fc012b5f3e55f7de570_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4586766448
<details>
<summary>TPC-H: <b>Total hot run time: 31393 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 749eb56236b404633b3b5fc012b5f3e55f7de570, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17663 4061 3996 3996
q2 q3 10757 1373 854 854
q4 4682 483 339 339
q5 7594 2236 2088 2088
q6 245 176 138 138
q7 904 804 638 638
q8 9482 1624 1531 1531
q9 5117 4973 4934 4934
q10 6382 2223 1843 1843
q11 438 272 241 241
q12 637 428 301 301
q13 18079 3359 2744 2744
q14 264 260 239 239
q15 q16 815 805 717 717
q17 933 899 917 899
q18 6920 5936 5478 5478
q19 1331 1189 1036 1036
q20 512 473 340 340
q21 6239 2873 2718 2718
q22 482 383 319 319
Total cold run time: 99476 ms
Total hot run time: 31393 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5047 4713 4784 4713
q2 q3 4921 5345 4660 4660
q4 2105 2206 1402 1402
q5 4779 4906 4662 4662
q6 227 174 128 128
q7 1918 1788 1580 1580
q8 2458 2120 2153 2120
q9 8070 7631 7405 7405
q10 4745 4714 4262 4262
q11 523 382 370 370
q12 735 741 537 537
q13 3011 3423 2834 2834
q14 284 289 261 261
q15 q16 689 702 603 603
q17 1282 1267 1251 1251
q18 7303 6854 6772 6772
q19 1128 1107 1085 1085
q20 2242 2229 1949 1949
q21 5298 4690 4496 4496
q22 527 461 426 426
Total cold run time: 57292 ms
Total hot run time: 51516 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4729208120
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4673240559
# FE Regression Coverage Report
Increment line coverage ` 20.24% (17/84)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60921_f798ef8d547cec261161ace7a3c6508994d20787_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60921_f798ef8d547cec261161ace7a3c6508994d20787_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4633399998
<details>
<summary>TPC-DS: <b>Total hot run time: 169151 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 052c6fd279371dfb63b1b5d048c7f082653ef511, data reload: false
query5 4343 634 486 486
query6 441 200 181 181
query7 4808 571 302 302
query8 372 228 203 203
query9 8761 3959 4016 3959
query10 437 327 261 261
query11 5791 2334 2234 2234
query12 159 105 98 98
query13 1243 577 464 464
query14 6405 5398 5017 5017
query14_1 4362 4352 4356 4352
query15 210 198 175 175
query16 1030 461 436 436
query17 1095 701 555 555
query18 2420 474 346 346
query19 199 180 141 141
query20 111 107 102 102
query21 209 140 116 116
query22 13723 13632 13418 13418
query23 17401 16500 16064 16064
query23_1 16291 16267 16533 16267
query24 7592 1776 1299 1299
query24_1 1308 1326 1313 1313
query25 547 441 388 388
query26 1318 322 159 159
query27 2657 551 341 341
query28 4464 2000 1994 1994
query29 1054 602 478 478
query30 302 235 197 197
query31 1129 1072 962 962
query32 112 60 60 60
query33 509 319 244 244
query34 1190 1107 661 661
query35 747 780 681 681
query36 1393 1412 1227 1227
query37 148 101 91 91
query38 3204 3131 3037 3037
query39 925 916 888 888
query39_1 889 902 874 874
query40 223 119 99 99
query41 63 61 60 60
query42 93 94 94 94
query43 317 318 278 278
query44
query45 194 181 177 177
query46 1049 1213 711 711
query47 2392 2356 2296 2296
query48 382 392 294 294
query49 630 508 378 378
query50 979 353 261 261
query51 4354 4262 4252 4252
query52 90 90 78 78
query53 253 274 198 198
query54 282 231 212 212
query55 81 77 72 72
query56 245 236 238 236
query57 1429 1412 1322 1322
query58 255 227 224 224
query59 1586 1699 1450 1450
query60 290 258 247 247
query61 186 184 180 180
query62 694 674 592 592
query63 232 191 187 187
query64 2610 848 676 676
query65
query66 1844 475 363 363
query67 29674 29707 29563 29563
query68
query69 425 319 269 269
query70 967 979 972 972
query71 319 229 215 215
query72 3259 2887 2564 2564
query73 826 731 414 414
query74 5169 4927 4757 4757
query75 2646 2552 2219 2219
query76 2307 1153 804 804
query77 349 378 305 305
query78 12399 12517 11866 11866
query79 1385 1064 755 755
query80 574 469 382 382
query81 452 280 243 243
query82 596 153 117 117
query83 362 271 253 253
query84 268 140 112 112
query85 887 526 459 459
query86 377 298 290 290
query87 3400 3402 3202 3202
query88 3620 2747 2714 2714
query89 444 372 328 328
query90 1992 178 181 178
query91 187 171 139 139
query92 64 59 59 59
query93 1473 1504 811 811
query94 546 349 313 313
query95 695 363 349 349
query96 1005 783 341 341
query97 2706 2736 2558 2558
query98 214 203 201 201
query99 1145 1172 1043 1043
Total cold run time: 251259 ms
Total hot run time: 169151 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4637881088
# FE Regression Coverage Report
Increment line coverage ` 7.23% (6/83)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60921_052c6fd279371dfb63b1b5d048c7f082653ef511_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60921_052c6fd279371dfb63b1b5d048c7f082653ef511_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4603209363
# FE Regression Coverage Report
Increment line coverage ` 16.52% (19/115)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60921_9172b9e209226839f6eaa2002c4d250013140767_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60921_9172b9e209226839f6eaa2002c4d250013140767_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4655356516
@CalvinKirs @morningman @gavinchou - Following-up for your assistance in getting the PR merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r3340033183
##########
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/manager/HttpUtils.java:
##########
@@ -60,18 +61,21 @@ public class HttpUtils {
static final int DEFAULT_TIME_OUT_MS = 2000;
public static List<Pair<String, Integer>> getFeList() {
+ int port = Config.enable_https ? Config.https_port : Config.http_port;
return Env.getCurrentEnv().getFrontends(null)
- .stream().filter(Frontend::isAlive).map(fe -> Pair.of(fe.getHost(), Config.http_port))
+ .stream().filter(Frontend::isAlive).map(fe -> Pair.of(fe.getHost(), port))
.collect(Collectors.toList());
}
static boolean isCurrentFe(String ip, int port) {
Review Comment:
@morningman Thanks for the review!
Just to clarify — isCurrentFe is actually used at QueryProfileAction.java:347, where it acts as a guard to skip the current FE node during the cluster-wide trace ID fan-out loop (to avoid the node forwarding a request to itself). Happy to add a more descriptive Javadoc if that would make it clearer.
https://github.com/apache/doris/blob/07b497ab2192a8eb3ccfe1e0c41a48123c646e1e/fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/manager/QueryProfileAction.java#L347
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4587239921
<details>
<summary>TPC-DS: <b>Total hot run time: 171575 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5c1f94c0bd3128aced58ce172868708101d080d7, data reload: false
query5 4338 669 516 516
query6 326 215 197 197
query7 4241 586 305 305
query8 325 250 218 218
query9 8767 4105 4064 4064
query10 452 343 299 299
query11 5823 2578 2237 2237
query12 182 128 121 121
query13 1279 628 455 455
query14 6081 5499 5112 5112
query14_1 4452 4521 4459 4459
query15 212 206 183 183
query16 983 482 420 420
query17 1124 708 596 596
query18 2446 483 346 346
query19 219 200 163 163
query20 133 136 132 132
query21 213 142 115 115
query22 13599 13730 13407 13407
query23 17250 16459 16268 16268
query23_1 16328 16246 16341 16246
query24 7465 1796 1307 1307
query24_1 1303 1342 1314 1314
query25 582 518 452 452
query26 1310 343 176 176
query27 2694 563 366 366
query28 4485 1997 2051 1997
query29 1052 649 535 535
query30 305 240 207 207
query31 1125 1094 995 995
query32 137 79 76 76
query33 558 369 308 308
query34 1168 1147 661 661
query35 781 827 739 739
query36 1455 1428 1344 1344
query37 159 114 99 99
query38 3275 3210 3157 3157
query39 965 976 938 938
query39_1 934 928 901 901
query40 241 156 132 132
query41 73 69 68 68
query42 113 114 112 112
query43 342 351 308 308
query44
query45 218 207 203 203
query46 1081 1217 770 770
query47 2323 2338 2152 2152
query48 414 411 305 305
query49 648 505 408 408
query50 962 362 253 253
query51 4436 4322 4275 4275
query52 110 109 98 98
query53 278 292 204 204
query54 336 295 280 280
query55 100 106 89 89
query56 326 314 318 314
query57 1433 1423 1333 1333
query58 319 297 281 281
query59 1582 1632 1472 1472
query60 342 346 312 312
query61 188 186 202 186
query62 699 648 587 587
query63 252 209 219 209
query64 2445 794 627 627
query65
query66 1750 482 356 356
query67 29864 29698 29602 29602
query68
query69 471 344 304 304
query70 1050 1014 932 932
query71 313 273 276 273
query72 2940 2706 2396 2396
query73 843 771 411 411
query74 5118 4927 4775 4775
query75 2709 2610 2252 2252
query76 2333 1173 779 779
query77 414 409 343 343
query78 12289 12474 11777 11777
query79 1489 1080 789 789
query80 1329 537 467 467
query81 529 280 239 239
query82 1307 161 122 122
query83 358 287 260 260
query84 266 145 114 114
query85 919 534 447 447
query86 454 344 312 312
query87 3435 3387 3211 3211
query88 3664 2755 2741 2741
query89 454 409 359 359
query90 1918 181 186 181
query91 221 171 139 139
query92 82 84 77 77
query93 1585 1473 865 865
query94 740 349 314 314
query95 680 491 375 375
query96 1079 788 338 338
query97 2740 2700 2606 2606
query98 238 228 225 225
query99 1177 1162 1024 1024
Total cold run time: 255497 ms
Total hot run time: 171575 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4634035636
Thanks @CalvinKirs for suggestion, same is updated with new commit - Please assist with approval and merging.
@morningman @gavinchou - a new commit removed approval, please help again with approval and merging this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-3996400587
<details>
<summary>TPC-DS: <b>Total hot run time: 184297 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bb0743447ea301486989c5534a1dc6b4a27e2071, data reload: false
query5 4331 641 530 530
query6 326 219 198 198
query7 4219 465 263 263
query8 333 242 245 242
query9 8692 2713 2760 2713
query10 520 367 330 330
query11 16991 17294 17092 17092
query12 183 130 121 121
query13 1496 491 357 357
query14 6814 3350 3176 3176
query14_1 2943 2997 2899 2899
query15 203 194 177 177
query16 1070 497 481 481
query17 1115 718 668 668
query18 2915 464 380 380
query19 238 211 185 185
query20 143 138 136 136
query21 214 154 124 124
query22 5142 4980 4786 4786
query23 17224 16795 16636 16636
query23_1 16766 16646 16682 16646
query24 7281 1619 1226 1226
query24_1 1215 1246 1236 1236
query25 533 449 416 416
query26 1233 264 149 149
query27 2778 466 278 278
query28 4560 1889 1877 1877
query29 796 551 486 486
query30 308 245 210 210
query31 870 723 644 644
query32 85 74 72 72
query33 525 341 288 288
query34 940 917 568 568
query35 633 734 605 605
query36 1096 1198 999 999
query37 143 98 85 85
query38 2984 3012 2852 2852
query39 900 871 842 842
query39_1 841 830 814 814
query40 238 160 144 144
query41 68 66 66 66
query42 108 106 105 105
query43 388 384 363 363
query44
query45 207 189 190 189
query46 881 1000 636 636
query47 2111 2135 2056 2056
query48 310 323 240 240
query49 630 474 410 410
query50 681 282 225 225
query51 4058 4039 4097 4039
query52 111 112 98 98
query53 293 359 292 292
query54 313 283 281 281
query55 98 87 83 83
query56 327 318 322 318
query57 1361 1332 1278 1278
query58 303 281 287 281
query59 2592 2714 2576 2576
query60 341 353 334 334
query61 180 173 171 171
query62 620 619 549 549
query63 321 280 292 280
query64 5058 1375 1078 1078
query65
query66 1462 516 384 384
query67 16304 16486 16323 16323
query68
query69 405 332 302 302
query70 1029 991 973 973
query71 341 316 307 307
query72 2800 2720 2645 2645
query73 546 552 320 320
query74 9946 9897 9778 9778
query75 2827 2778 2482 2482
query76 2315 1063 662 662
query77 360 383 313 313
query78 11141 11290 10641 10641
query79 3075 812 608 608
query80 1760 654 533 533
query81 585 282 247 247
query82 995 151 116 116
query83 332 261 245 245
query84 248 113 95 95
query85 919 476 426 426
query86 447 298 306 298
query87 3079 3121 2976 2976
query88 3560 2658 2637 2637
query89 434 373 351 351
query90 2002 175 166 166
query91 168 162 136 136
query92 80 76 73 73
query93 1386 836 510 510
query94 645 324 296 296
query95 589 346 380 346
query96 647 512 227 227
query97 2478 2523 2395 2395
query98 242 220 211 211
query99 1018 1017 886 886
Total cold run time: 256755 ms
Total hot run time: 184297 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-3996341259
<details>
<summary>TPC-H: <b>Total hot run time: 29027 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bb0743447ea301486989c5534a1dc6b4a27e2071, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17696 4435 4267 4267
q2 q3 10640 826 516 516
q4 4690 378 256 256
q5 7587 1229 1043 1043
q6 184 188 157 157
q7 806 890 681 681
q8 9331 1511 1357 1357
q9 4959 4754 4749 4749
q10 6830 1889 1652 1652
q11 490 266 247 247
q12 767 570 470 470
q13 17772 4221 3443 3443
q14 240 230 221 221
q15 964 802 801 801
q16 765 722 666 666
q17 720 842 441 441
q18 6049 5397 5228 5228
q19 1178 980 619 619
q20 534 506 396 396
q21 4761 2074 1535 1535
q22 376 335 282 282
Total cold run time: 97339 ms
Total hot run time: 29027 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4603 4587 4621 4587
q2 q3 1890 2212 1788 1788
q4 841 1192 760 760
q5 4063 4368 4462 4368
q6 194 185 142 142
q7 1748 1653 1542 1542
q8 2496 2704 2590 2590
q9 7528 7498 7385 7385
q10 2628 2852 2409 2409
q11 508 442 415 415
q12 501 604 543 543
q13 3977 4384 3557 3557
q14 274 296 283 283
q15 864 917 962 917
q16 728 778 719 719
q17 1192 1506 1370 1370
q18 7161 7016 6547 6547
q19 863 872 865 865
q20 2124 2161 2025 2025
q21 3922 3472 3468 3468
q22 449 438 381 381
Total cold run time: 48554 ms
Total hot run time: 46661 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-3997119921
# FE Regression Coverage Report
Increment line coverage ` 15.73% (14/89)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60921_bb0743447ea301486989c5534a1dc6b4a27e2071_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60921_bb0743447ea301486989c5534a1dc6b4a27e2071_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-3985857934
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-3986030069
<details>
<summary>TPC-H: <b>Total hot run time: 28728 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 29f45dbe504c8d46be0de681c14f24d3a97c8291, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17684 4460 4289 4289
q2 q3 10648 764 517 517
q4 4680 362 249 249
q5 7562 1191 1015 1015
q6 178 176 149 149
q7 799 835 682 682
q8 9311 1426 1316 1316
q9 4802 4758 4692 4692
q10 6803 1857 1651 1651
q11 476 269 232 232
q12 718 566 470 470
q13 17774 4199 3412 3412
q14 245 236 212 212
q15 928 805 799 799
q16 770 716 681 681
q17 731 857 419 419
q18 5889 5411 5196 5196
q19 1178 964 601 601
q20 507 492 392 392
q21 4756 1996 1453 1453
q22 390 341 301 301
Total cold run time: 96829 ms
Total hot run time: 28728 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4762 4651 4592 4592
q2 q3 1828 2177 1772 1772
q4 879 1187 794 794
q5 4012 4336 4277 4277
q6 180 186 140 140
q7 1770 1634 1517 1517
q8 2488 2690 2508 2508
q9 7478 7402 7416 7402
q10 2622 2818 2368 2368
q11 533 459 437 437
q12 515 611 473 473
q13 4098 4792 3592 3592
q14 283 302 273 273
q15 868 841 802 802
q16 712 755 714 714
q17 1156 1670 1290 1290
q18 7183 6960 6655 6655
q19 931 1009 930 930
q20 2072 2193 1988 1988
q21 3960 3439 3280 3280
q22 462 432 373 373
Total cold run time: 48792 ms
Total hot run time: 46177 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r3427648933
##########
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/manager/HttpUtils.java:
##########
@@ -130,8 +135,11 @@ public static CloseableHttpClient getHttpClient() {
}
private static String executeRequest(HttpRequestBase request) throws IOException {
- CloseableHttpClient client = getHttpClient();
- return client.execute(request, httpResponse -> EntityUtils.toString(httpResponse.getEntity()));
+ try (CloseableHttpClient client = Config.enable_https
Review Comment:
This should choose the HTTPS-aware client from the request URI scheme, not only from the global FE HTTPS flag. `HttpUtils` still drives plain BE calls: `configurationName()` builds `http://<be>/api/show_config`, `handleConfigurationInfo(..., "BE", ...)` forces `http://`, and `concatBeSetConfigUrl()` builds `http://.../api/update_config`. With `Config.enable_https=true`, those BE calls now enter `InternalHttpsUtils.createValidatedHttpClient()`, which loads `mysql_ssl_default_ca_certificate` before executing the request. A deployment can enable FE HTTPS while not relying on that MySQL CA path for BE HTTP calls, so plain BE manager/config requests can fail before connecting to the BE. Please gate the internal HTTPS client on `"https".equalsIgnoreCase(request.getURI().getScheme()) && Config.enable_https`, and keep the plain client for `http://` URLs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4850001714
We can also merge this first and follow up with a separate PR to narrow the HttpUtils/InternalHttpsUtils scope if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #60921:
URL: https://github.com/apache/doris/pull/60921#discussion_r3379302988
##########
fe/fe-core/src/main/java/org/apache/doris/system/HeartbeatMgr.java:
##########
@@ -96,7 +97,7 @@ public void setMaster(int clusterId, String token, long epoch) {
TMasterInfo tMasterInfo = new TMasterInfo(
new TNetworkAddress(FrontendOptions.getLocalHostAddress(), Config.rpc_port), clusterId, epoch);
tMasterInfo.setToken(token);
- tMasterInfo.setHttpPort(Config.http_port);
+ tMasterInfo.setHttpPort(HttpURLUtil.getHttpPort());
long flags = heartbeatFlags.getHeartbeatFlags();
tMasterInfo.setHeartbeatFlags(flags);
Review Comment:
Sending `https_port` here breaks the current BE consumer of `TMasterInfo.http_port`. BE stores this as `ClusterInfo::master_fe_http_port` in `heartbeat_server.cpp`, and `SmallFileMgr::_download_file()` builds `master_fe_addr.hostname:master_fe_http_port/api/get_small_file?...` and calls `HttpClient::init()` without adding an `https://` scheme or configuring the FE truststore. With `enable_https=true`, BEs will attempt the existing plain-HTTP small-file download against the FE HTTPS listener and fail. Until the BE path is updated to understand HTTPS and trust configuration, this heartbeat field should continue to carry the FE plain HTTP port, or a separate scheme/HTTPS-aware field/path should be added and consumed end-to-end.
##########
fe/fe-core/src/main/java/org/apache/doris/httpv2/meta/MetaService.java:
##########
@@ -163,8 +163,7 @@ public Object put(HttpServletRequest request, HttpServletResponse response) thro
clientHost, clientPort, machine, portStr);
clientHost = Strings.isNullOrEmpty(clientHost) ? machine : clientHost;
- String url = "http://" + NetUtils.getHostPortInAccessibleFormat(clientHost, Integer.valueOf(portStr))
- + "/image?version=" + versionStr;
+ String url = HttpURLUtil.buildInternalFeUrl(clientHost, "/image", "version=" + versionStr);
Review Comment:
This drops the callback port supplied by the master in `/put?port=...`. The receiver still validates `portStr`, but now ignores it and uses its own local `Config.http_port`/`https_port` via `buildInternalFeUrl()`. In the checkpoint flow the master sends its reachable image port at `Checkpoint.doCheckpoint()`; if FE HTTP/HTTPS ports differ across nodes during a port override/rolling change, the follower will try to fetch `/image` from the master using the follower's local port instead of the master-provided port and image push fails. Please keep using the validated `portStr` when building this callback URL, with only the scheme switched for HTTPS if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement] (FE) Convert intra FE-to-FE calls to HTTPS when enabled [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #60921:
URL: https://github.com/apache/doris/pull/60921#issuecomment-4689275567
@morrySnow please could you help with a review and merging this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org