You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "haifxu (via GitHub)" <gi...@apache.org> on 2023/03/17 02:53:08 UTC
[GitHub] [inlong] haifxu opened a new pull request, #7629: [INLONG-7413][Audit] Audit get MQ config from Manager
haifxu opened a new pull request, #7629:
URL: https://github.com/apache/inlong/pull/7629
### Prepare a Pull Request
- Fixes #7413
### Motivation
Get the MQ cluster address from the manager.
### Modifications
1. Fix `httpGet` to `httpPost`
2. Add managerHosts and cluster params in `server.properties`
### Verifying this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #7629: [INLONG-7413][Audit] Audit get MQ config from Manager
Posted by "healchow (via GitHub)" <gi...@apache.org>.
healchow commented on code in PR #7629:
URL: https://github.com/apache/inlong/pull/7629#discussion_r1139822215
##########
inlong-audit/audit-store/src/main/java/org/apache/inlong/audit/service/AuditMsgConsumerServer.java:
##########
@@ -96,4 +127,66 @@ private List<InsertData> getInsertServiceList() {
}
return insertServiceList;
}
+
+ private List<MQClusterInfo> getConfigManager() {
+ Properties properties = new Properties();
+ try (InputStream inputStream = getClass().getClassLoader().getResourceAsStream(DEFAULT_CONFIG_PROPERTIES)) {
+ properties.load(inputStream);
+ String managerHosts = properties.getProperty("manager.hosts");
+ String clusterName = properties.getProperty("proxy.cluster.name");
+ String clusterTag = properties.getProperty("proxy.cluster.tag");
+ String[] hostList = StringUtils.split(managerHosts, ",");
+ for (String host : hostList) {
+ List<MQClusterInfo> mqConfig = getMQConfig(host, clusterName, clusterTag);
+ if (ObjectUtils.isNotEmpty(mqConfig)) {
+ return mqConfig;
+ }
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ return null;
+ }
+
+ private List<MQClusterInfo> getMQConfig(String host, String clusterName, String clusterTag) {
+ HttpPost httpPost = null;
+ Gson gson = new Gson();
+ try {
+ String url = "http://" + host + "/inlong/manager/openapi/dataproxy/getConfig";
+ LOG.info("start to request {} to get config info", url);
+ httpPost = new HttpPost(url);
+ httpPost.addHeader(HttpHeaders.CONNECTION, "close");
+
+ // request body
+ DataProxyConfigRequest request = new DataProxyConfigRequest();
+ request.setClusterName(clusterName);
+ request.setClusterTag(clusterTag);
+ StringEntity stringEntity = new StringEntity(gson.toJson(request));
+ stringEntity.setContentType("application/json");
+ httpPost.setEntity(stringEntity);
+
+ // request with post
+ LOG.info("start to request {} to get config info with params {}", url, request);
+ CloseableHttpResponse response = httpClient.execute(httpPost);
+ String returnStr = EntityUtils.toString(response.getEntity());
+ // get groupId <-> topic and m value.
+
+ RemoteConfigJson configJson = gson.fromJson(returnStr, RemoteConfigJson.class);
+ if (configJson.isSuccess() && configJson.getData() != null) {
+ LOG.info("getConfig result: {}", configJson);
+ List<MQClusterInfo> mqClusterInfoList = configJson.getData().getMqClusterList();
+ if (mqClusterInfoList != null && !mqClusterInfoList.isEmpty()) {
+ return mqClusterInfoList;
+ }
+ }
+ } catch (Exception ex) {
+ LOG.error("exception caught", ex);
Review Comment:
Please add more useful info in error log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #7629: [INLONG-7413][Audit] Audit get MQ config from Manager
Posted by "healchow (via GitHub)" <gi...@apache.org>.
healchow commented on code in PR #7629:
URL: https://github.com/apache/inlong/pull/7629#discussion_r1139824371
##########
inlong-audit/audit-store/src/main/java/org/apache/inlong/audit/service/AuditMsgConsumerServer.java:
##########
@@ -96,4 +127,66 @@ private List<InsertData> getInsertServiceList() {
}
return insertServiceList;
}
+
+ private List<MQClusterInfo> getConfigManager() {
Review Comment:
`getConfigManager` -> `getClusterListFromManager` maybe more clearly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #7629: [INLONG-7413][Audit] Audit get MQ config from Manager
Posted by "healchow (via GitHub)" <gi...@apache.org>.
healchow commented on code in PR #7629:
URL: https://github.com/apache/inlong/pull/7629#discussion_r1139821609
##########
inlong-audit/audit-store/src/main/java/org/apache/inlong/audit/service/AuditMsgConsumerServer.java:
##########
@@ -96,4 +127,66 @@ private List<InsertData> getInsertServiceList() {
}
return insertServiceList;
}
+
+ private List<MQClusterInfo> getConfigManager() {
+ Properties properties = new Properties();
+ try (InputStream inputStream = getClass().getClassLoader().getResourceAsStream(DEFAULT_CONFIG_PROPERTIES)) {
+ properties.load(inputStream);
+ String managerHosts = properties.getProperty("manager.hosts");
+ String clusterName = properties.getProperty("proxy.cluster.name");
+ String clusterTag = properties.getProperty("proxy.cluster.tag");
+ String[] hostList = StringUtils.split(managerHosts, ",");
+ for (String host : hostList) {
+ List<MQClusterInfo> mqConfig = getMQConfig(host, clusterName, clusterTag);
+ if (ObjectUtils.isNotEmpty(mqConfig)) {
+ return mqConfig;
+ }
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ return null;
+ }
+
+ private List<MQClusterInfo> getMQConfig(String host, String clusterName, String clusterTag) {
+ HttpPost httpPost = null;
+ Gson gson = new Gson();
Review Comment:
Suggest use a global Gson instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #7629: [INLONG-7413][Audit] Audit get MQ config from Manager
Posted by "healchow (via GitHub)" <gi...@apache.org>.
healchow commented on code in PR #7629:
URL: https://github.com/apache/inlong/pull/7629#discussion_r1139830670
##########
inlong-audit/audit-common/src/main/java/org/apache/inlong/audit/file/ConfigManager.java:
##########
@@ -189,33 +200,42 @@ private void checkLocalFile() {
}
}
- private boolean checkWithManager(String host) {
- HttpGet httpGet = null;
+ private boolean checkWithManager(String host, String clusterName, String clusterTag) {
+ HttpPost httpPost = null;
try {
- String url = "http://" + host + "/inlong/manager/openapi/audit/getConfig";
+ String url = "http://" + host + "/inlong/manager/openapi/dataproxy/getConfig";
Review Comment:
Suggest extract to a constant, or read from the config file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] haifxu commented on a diff in pull request #7629: [INLONG-7413][Audit] Audit get MQ config from Manager
Posted by "haifxu (via GitHub)" <gi...@apache.org>.
haifxu commented on code in PR #7629:
URL: https://github.com/apache/inlong/pull/7629#discussion_r1139925393
##########
inlong-audit/audit-store/src/main/java/org/apache/inlong/audit/service/AuditMsgConsumerServer.java:
##########
@@ -96,4 +127,66 @@ private List<InsertData> getInsertServiceList() {
}
return insertServiceList;
}
+
+ private List<MQClusterInfo> getConfigManager() {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #7629: [INLONG-7413][Audit] Audit get MQ config from Manager
Posted by "healchow (via GitHub)" <gi...@apache.org>.
healchow commented on code in PR #7629:
URL: https://github.com/apache/inlong/pull/7629#discussion_r1139823959
##########
inlong-audit/audit-store/src/main/java/org/apache/inlong/audit/service/AuditMsgConsumerServer.java:
##########
@@ -96,4 +127,66 @@ private List<InsertData> getInsertServiceList() {
}
return insertServiceList;
}
+
+ private List<MQClusterInfo> getConfigManager() {
+ Properties properties = new Properties();
+ try (InputStream inputStream = getClass().getClassLoader().getResourceAsStream(DEFAULT_CONFIG_PROPERTIES)) {
+ properties.load(inputStream);
+ String managerHosts = properties.getProperty("manager.hosts");
+ String clusterName = properties.getProperty("proxy.cluster.name");
+ String clusterTag = properties.getProperty("proxy.cluster.tag");
+ String[] hostList = StringUtils.split(managerHosts, ",");
+ for (String host : hostList) {
+ List<MQClusterInfo> mqConfig = getMQConfig(host, clusterName, clusterTag);
+ if (ObjectUtils.isNotEmpty(mqConfig)) {
+ return mqConfig;
+ }
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ return null;
+ }
+
+ private List<MQClusterInfo> getMQConfig(String host, String clusterName, String clusterTag) {
+ HttpPost httpPost = null;
+ Gson gson = new Gson();
+ try {
+ String url = "http://" + host + "/inlong/manager/openapi/dataproxy/getConfig";
Review Comment:
Extract to a global contant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] healchow commented on a diff in pull request #7629: [INLONG-7413][Audit] Audit get MQ config from Manager
Posted by "healchow (via GitHub)" <gi...@apache.org>.
healchow commented on code in PR #7629:
URL: https://github.com/apache/inlong/pull/7629#discussion_r1139822488
##########
inlong-audit/audit-store/src/main/java/org/apache/inlong/audit/service/AuditMsgConsumerServer.java:
##########
@@ -96,4 +127,66 @@ private List<InsertData> getInsertServiceList() {
}
return insertServiceList;
}
+
+ private List<MQClusterInfo> getConfigManager() {
+ Properties properties = new Properties();
+ try (InputStream inputStream = getClass().getClassLoader().getResourceAsStream(DEFAULT_CONFIG_PROPERTIES)) {
+ properties.load(inputStream);
+ String managerHosts = properties.getProperty("manager.hosts");
+ String clusterName = properties.getProperty("proxy.cluster.name");
+ String clusterTag = properties.getProperty("proxy.cluster.tag");
+ String[] hostList = StringUtils.split(managerHosts, ",");
+ for (String host : hostList) {
+ List<MQClusterInfo> mqConfig = getMQConfig(host, clusterName, clusterTag);
+ if (ObjectUtils.isNotEmpty(mqConfig)) {
+ return mqConfig;
+ }
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ return null;
+ }
+
+ private List<MQClusterInfo> getMQConfig(String host, String clusterName, String clusterTag) {
+ HttpPost httpPost = null;
+ Gson gson = new Gson();
+ try {
+ String url = "http://" + host + "/inlong/manager/openapi/dataproxy/getConfig";
+ LOG.info("start to request {} to get config info", url);
+ httpPost = new HttpPost(url);
+ httpPost.addHeader(HttpHeaders.CONNECTION, "close");
+
+ // request body
+ DataProxyConfigRequest request = new DataProxyConfigRequest();
+ request.setClusterName(clusterName);
+ request.setClusterTag(clusterTag);
+ StringEntity stringEntity = new StringEntity(gson.toJson(request));
+ stringEntity.setContentType("application/json");
+ httpPost.setEntity(stringEntity);
+
+ // request with post
+ LOG.info("start to request {} to get config info with params {}", url, request);
+ CloseableHttpResponse response = httpClient.execute(httpPost);
+ String returnStr = EntityUtils.toString(response.getEntity());
+ // get groupId <-> topic and m value.
+
Review Comment:
Unnecessary blank line here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] dockerzhang merged pull request #7629: [INLONG-7413][Audit] Audit get MQ config from Manager
Posted by "dockerzhang (via GitHub)" <gi...@apache.org>.
dockerzhang merged PR #7629:
URL: https://github.com/apache/inlong/pull/7629
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org