You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@streampark.apache.org by "xujiangfeng001 (via GitHub)" <gi...@apache.org> on 2023/04/21 08:45:33 UTC
[GitHub] [incubator-streampark] xujiangfeng001 opened a new pull request, #2675: [ISSUE-2498][Feature] [SubTask] The cluster supports remote and yarn session heartbeat monitoring
xujiangfeng001 opened a new pull request, #2675:
URL: https://github.com/apache/incubator-streampark/pull/2675
<!--
Thank you for contributing to StreamPark! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
## Contribution Checklist
- If this is your first time, please read our contributor guidelines: [Submit Code](https://streampark.apache.org/community/submit_guide/submit_code).
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-streampark/issues).
- Name the pull request in the form "[Feature] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- If the PR is unfinished, add `[WIP]` in your PR title, e.g., `[WIP][Feature] Title of the pull request`.
-->
## What changes were proposed in this pull request
Issue Number: https://github.com/apache/incubator-streampark/issues/2498
This PR is a resubmission of https://github.com/apache/incubator-streampark/pull/2541. Compared to https://github.com/apache/incubator-streampark/pull/2541, this PR modifies the state change as follows:
Cluster status:
- CREATED
- RUNNING
- STOPPED
- LOST
- UNKNOWN
If the state obtained from Yarn is UNDEFINED, it will be mapped to RUNNING of cluster status. If the states obtained from Yarn are FAILED, KILLED, SUCCEEDED and ENDED, they will be mapped as STOPPED for cluster status.
This pr supports status monitoring and status updates for the "Flink Cluster on Yarn" and "Remote".Heartbeat and status monitoring support for "Flink Cluster on K8s" will be supported in subsequent pr support.
## Brief change log
Nothing.
## Verifying this change
This change without any test coverage.
## Does this pull request potentially affect one of the following parts
- Dependencies (does it add or upgrade a dependency): no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] xujiangfeng001 commented on a diff in pull request #2675: [ISSUE-2498][Feature] [SubTask] The cluster supports remote and yarn session heartbeat monitoring
Posted by "xujiangfeng001 (via GitHub)" <gi...@apache.org>.
xujiangfeng001 commented on code in PR #2675:
URL: https://github.com/apache/incubator-streampark/pull/2675#discussion_r1201966802
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/FlinkClusterServiceImpl.java:
##########
@@ -140,12 +141,16 @@ public Boolean create(FlinkCluster flinkCluster) {
ApiAlertException.throwIfFalse(
successful, String.format(ERROR_CLUSTER_QUEUE_HINT, flinkCluster.getYarnQueue()));
flinkCluster.setCreateTime(new Date());
- if (ExecutionMode.isRemoteMode(flinkCluster.getExecutionModeEnum())) {
- flinkCluster.setClusterState(ClusterState.STARTED.getValue());
+ if (ExecutionMode.REMOTE.equals(flinkCluster.getExecutionModeEnum())) {
Review Comment:
> ExecutionMode.isRemote(...
Thank you very much for the suggestion. It's a good suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on pull request #2675: [ISSUE-2498][Feature] [SubTask] The cluster supports remote and yarn session heartbeat monitoring
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #2675:
URL: https://github.com/apache/incubator-streampark/pull/2675#issuecomment-1518639081
> Hi @RocMarshal ,This is my resubmitted PR for #2541. This PR has modified some suggestions for #2541.
>
> For the issue of [#2541 (comment)](https://github.com/apache/incubator-streampark/pull/2541#discussion_r1162910882), I have thought about it for a moment. In fact, it will not have an impact on obtaining the cluster status because the cluster failed to obtain the corresponding status information from` jobManagerUrl` and will go back to Yarn.
>
> Looking forward to your next review.
Thank you @xujiangfeng001 very much for your update. It seems closer to the ideal state.
Due to my personal reasons, the review may be conducted later.
Other reviewers CC @wolfboys @macksonmu can also participate in this MR review.
If you have time on your end, you can try to continue to advance the design and discussion of other subtasks of https://github.com/apache/incubator-streampark/issues/2425
which will shorten some of the connection time~
Best Roc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2675: [ISSUE-2498][Feature] [SubTask] The cluster supports remote and yarn session heartbeat monitoring
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2675:
URL: https://github.com/apache/incubator-streampark/pull/2675#discussion_r1199555129
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/FlinkClusterServiceImpl.java:
##########
@@ -140,12 +141,16 @@ public Boolean create(FlinkCluster flinkCluster) {
ApiAlertException.throwIfFalse(
successful, String.format(ERROR_CLUSTER_QUEUE_HINT, flinkCluster.getYarnQueue()));
flinkCluster.setCreateTime(new Date());
- if (ExecutionMode.isRemoteMode(flinkCluster.getExecutionModeEnum())) {
- flinkCluster.setClusterState(ClusterState.STARTED.getValue());
+ if (ExecutionMode.REMOTE.equals(flinkCluster.getExecutionModeEnum())) {
Review Comment:
ExecutionMode.isRemote(...
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/FlinkClusterWatcher.java:
##########
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.streampark.console.core.task;
+
+import org.apache.streampark.common.enums.ClusterState;
+import org.apache.streampark.common.enums.ExecutionMode;
+import org.apache.streampark.common.util.HttpClientUtils;
+import org.apache.streampark.common.util.ThreadUtils;
+import org.apache.streampark.common.util.YarnUtils;
+import org.apache.streampark.console.base.util.JacksonUtils;
+import org.apache.streampark.console.core.entity.FlinkCluster;
+import org.apache.streampark.console.core.metrics.flink.Overview;
+import org.apache.streampark.console.core.metrics.yarn.YarnAppInfo;
+import org.apache.streampark.console.core.service.FlinkClusterService;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hc.client5.http.config.RequestConfig;
+
+import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
+import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.scheduling.annotation.Scheduled;
+import org.springframework.stereotype.Component;
+
+import javax.annotation.PostConstruct;
+import javax.annotation.PreDestroy;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/** This implementation is currently used for tracing Cluster on yarn,remote,K8s mode */
+@Slf4j
+@Component
+public class FlinkClusterWatcher {
+
+ @Autowired private FlinkClusterService flinkClusterService;
+
+ private Long lastWatcheringTime = 0L;
+
+ // Track interval every 30 seconds
+ private static final Duration WATCHER_INTERVAL = Duration.ofSeconds(30);
+
+ /** Watcher cluster lists */
+ private static final Map<Long, FlinkCluster> WATCHER_CLUSTERS = new ConcurrentHashMap<>(0);
Review Comment:
```suggestion
private static final Map<Long, FlinkCluster> WATCHER_CLUSTERS = new ConcurrentHashMap<>(8);
```
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/FlinkClusterServiceImpl.java:
##########
@@ -160,16 +165,19 @@ public void start(FlinkCluster cluster) {
if (ExecutionMode.isYarnSessionMode(flinkCluster.getExecutionModeEnum())) {
String address = YarnUtils.getRMWebAppURL() + "/proxy/" + deployResponse.clusterId() + "/";
flinkCluster.setAddress(address);
+ flinkCluster.setJobManagerUrl(deployResponse.address());
} else {
flinkCluster.setAddress(deployResponse.address());
}
flinkCluster.setClusterId(deployResponse.clusterId());
- flinkCluster.setClusterState(ClusterState.STARTED.getValue());
+ flinkCluster.setClusterState(ClusterState.RUNNING.getValue());
flinkCluster.setException(null);
updateById(flinkCluster);
FlinkRESTAPIWatcher.removeFlinkCluster(flinkCluster);
Review Comment:
if the cluster is started successfully here, then we should add it into the monitor ?
In other scenerios with my limited read , `update`, `delete`,`shutdown` `start`, Its should be processed the logic by using FlinkClusterWatcher.addFlinkCluster or removeFlinkCluster` if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] xujiangfeng001 commented on pull request #2675: [ISSUE-2498][Feature] [SubTask] The cluster supports remote and yarn session heartbeat monitoring
Posted by "xujiangfeng001 (via GitHub)" <gi...@apache.org>.
xujiangfeng001 commented on PR #2675:
URL: https://github.com/apache/incubator-streampark/pull/2675#issuecomment-1566625662
> thanks for the update. LGTM +1. non-binding.
Thank you for your serious review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys merged pull request #2675: [ISSUE-2498][Feature] [SubTask] The cluster supports remote and yarn session heartbeat monitoring
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys merged PR #2675:
URL: https://github.com/apache/incubator-streampark/pull/2675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] xujiangfeng001 commented on pull request #2675: [ISSUE-2498][Feature] [SubTask] The cluster supports remote and yarn session heartbeat monitoring
Posted by "xujiangfeng001 (via GitHub)" <gi...@apache.org>.
xujiangfeng001 commented on PR #2675:
URL: https://github.com/apache/incubator-streampark/pull/2675#issuecomment-1568460293
> We need move the upgraded SQL to 2.2.0.sql file
Thank you very much for your reminder. I have modified the upgraded SQL to 2.2.0.sql. Looking forward to your next review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] xujiangfeng001 commented on a diff in pull request #2675: [ISSUE-2498][Feature] [SubTask] The cluster supports remote and yarn session heartbeat monitoring
Posted by "xujiangfeng001 (via GitHub)" <gi...@apache.org>.
xujiangfeng001 commented on code in PR #2675:
URL: https://github.com/apache/incubator-streampark/pull/2675#discussion_r1201982143
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/FlinkClusterServiceImpl.java:
##########
@@ -160,16 +165,19 @@ public void start(FlinkCluster cluster) {
if (ExecutionMode.isYarnSessionMode(flinkCluster.getExecutionModeEnum())) {
String address = YarnUtils.getRMWebAppURL() + "/proxy/" + deployResponse.clusterId() + "/";
flinkCluster.setAddress(address);
+ flinkCluster.setJobManagerUrl(deployResponse.address());
} else {
flinkCluster.setAddress(deployResponse.address());
}
flinkCluster.setClusterId(deployResponse.clusterId());
- flinkCluster.setClusterState(ClusterState.STARTED.getValue());
+ flinkCluster.setClusterState(ClusterState.RUNNING.getValue());
flinkCluster.setException(null);
updateById(flinkCluster);
FlinkRESTAPIWatcher.removeFlinkCluster(flinkCluster);
Review Comment:
> if the cluster is started successfully here, then we should add it into the monitor ?
>
> In other scenerios with my limited read , `update`, `delete`,`shutdown` `start`, Its should be processed the logic by using FlinkClusterWatcher.addFlinkCluster or removeFlinkCluster` if needed.
This is my negligence. It is indeed necessary to remove it from the cache during `shutdown`, `start`.As for the `update`and `delete`scenarios, as the `flink cluster` is stopped, it means that there is no longer any `flink cluster` information in the cache, so I don't think it needs to be removed from the cache anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] xujiangfeng001 commented on pull request #2675: [ISSUE-2498][Feature] [SubTask] The cluster supports remote and yarn session heartbeat monitoring
Posted by "xujiangfeng001 (via GitHub)" <gi...@apache.org>.
xujiangfeng001 commented on PR #2675:
URL: https://github.com/apache/incubator-streampark/pull/2675#issuecomment-1558983286
> Hi,, @xujiangfeng001 Sorry for the late review & Thanks for your efforts on for items mentioned in the previous PR.
>
> I left a few of comments, Please let me know what's your opnion.
Hi @RocMarshal ，thank you very much for your suggestion. I have replied and made modifications to the relevant content. Looking forward to your next review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] xujiangfeng001 commented on pull request #2675: [ISSUE-2498][Feature] [SubTask] The cluster supports remote and yarn session heartbeat monitoring
Posted by "xujiangfeng001 (via GitHub)" <gi...@apache.org>.
xujiangfeng001 commented on PR #2675:
URL: https://github.com/apache/incubator-streampark/pull/2675#issuecomment-1517502914
Hi @RocMarshal ,This is my resubmitted PR for https://github.com/apache/incubator-streampark/pull/2541. This PR has modified some suggestions for https://github.com/apache/incubator-streampark/pull/2541.
For the issue of https://github.com/apache/incubator-streampark/pull/2541#discussion_r1162910882, I have thought about it for a moment. In fact, it will not have an impact on obtaining the cluster status because the cluster failed to obtain the corresponding status information from` jobManagerUrl` and will go back to Yarn.
Looking forward to your next review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] xujiangfeng001 commented on a diff in pull request #2675: [ISSUE-2498][Feature] [SubTask] The cluster supports remote and yarn session heartbeat monitoring
Posted by "xujiangfeng001 (via GitHub)" <gi...@apache.org>.
xujiangfeng001 commented on code in PR #2675:
URL: https://github.com/apache/incubator-streampark/pull/2675#discussion_r1201965610
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/FlinkClusterServiceImpl.java:
##########
@@ -140,12 +141,16 @@ public Boolean create(FlinkCluster flinkCluster) {
ApiAlertException.throwIfFalse(
successful, String.format(ERROR_CLUSTER_QUEUE_HINT, flinkCluster.getYarnQueue()));
flinkCluster.setCreateTime(new Date());
- if (ExecutionMode.isRemoteMode(flinkCluster.getExecutionModeEnum())) {
- flinkCluster.setClusterState(ClusterState.STARTED.getValue());
+ if (ExecutionMode.REMOTE.equals(flinkCluster.getExecutionModeEnum())) {
Review Comment:
Thank you very much for the suggestion. It's a good suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] xujiangfeng001 commented on a diff in pull request #2675: [ISSUE-2498][Feature] [SubTask] The cluster supports remote and yarn session heartbeat monitoring
Posted by "xujiangfeng001 (via GitHub)" <gi...@apache.org>.
xujiangfeng001 commented on code in PR #2675:
URL: https://github.com/apache/incubator-streampark/pull/2675#discussion_r1201967266
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/FlinkClusterWatcher.java:
##########
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.streampark.console.core.task;
+
+import org.apache.streampark.common.enums.ClusterState;
+import org.apache.streampark.common.enums.ExecutionMode;
+import org.apache.streampark.common.util.HttpClientUtils;
+import org.apache.streampark.common.util.ThreadUtils;
+import org.apache.streampark.common.util.YarnUtils;
+import org.apache.streampark.console.base.util.JacksonUtils;
+import org.apache.streampark.console.core.entity.FlinkCluster;
+import org.apache.streampark.console.core.metrics.flink.Overview;
+import org.apache.streampark.console.core.metrics.yarn.YarnAppInfo;
+import org.apache.streampark.console.core.service.FlinkClusterService;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
+import org.apache.hc.client5.http.config.RequestConfig;
+
+import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
+import com.baomidou.mybatisplus.core.conditions.update.LambdaUpdateWrapper;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.scheduling.annotation.Scheduled;
+import org.springframework.stereotype.Component;
+
+import javax.annotation.PostConstruct;
+import javax.annotation.PreDestroy;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+/** This implementation is currently used for tracing Cluster on yarn,remote,K8s mode */
+@Slf4j
+@Component
+public class FlinkClusterWatcher {
+
+ @Autowired private FlinkClusterService flinkClusterService;
+
+ private Long lastWatcheringTime = 0L;
+
+ // Track interval every 30 seconds
+ private static final Duration WATCHER_INTERVAL = Duration.ofSeconds(30);
+
+ /** Watcher cluster lists */
+ private static final Map<Long, FlinkCluster> WATCHER_CLUSTERS = new ConcurrentHashMap<>(0);
Review Comment:
>
Thank you very much for the suggestion. It's a good suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on pull request #2675: [ISSUE-2498][Feature] [SubTask] The cluster supports remote and yarn session heartbeat monitoring
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #2675:
URL: https://github.com/apache/incubator-streampark/pull/2675#issuecomment-1564500757
thanks for the update.
LGTM +1. non-binding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org