You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@streampark.apache.org by "RocMarshal (via GitHub)" <gi...@apache.org> on 2023/01/28 11:31:42 UTC
[GitHub] [incubator-streampark] RocMarshal opened a new pull request, #2268: [Feature][Issue-2192] Support trigger savepoint manually.
RocMarshal opened a new pull request, #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268
<!--
Thank you for contributing to StreamPark! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
## Contribution Checklist
- If this is your first time, please read our contributor guidelines: [Submit Code](https://streampark.apache.org/community/submit_guide/submit_code).
- Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/incubator-streampark/issues).
- Name the pull request in the form "[Feature] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- If the PR is unfinished, add `[WIP]` in your PR title, e.g., `[WIP][Feature] Title of the pull request`.
-->
## What changes were proposed in this pull request
Issue Number: close #2192 <!-- REMOVE this line if no issue to close -->
<!--(For example: This pull request proposed to add checkstyle plugin).-->
## Brief change log
- Backend
- Add the triggerSavepoint interface;
- Adapt the new colums fill logic for corresponding code segment;
- Add the savepoint triggered by users store logic.
- Minor refactor for `CheckpointProcessor.java`
- FE
Add the savepoint icon-button and the corresponding logic about calling triggerSavepoint interface to backend.
## Verifying this change
- *Manually verified the change by testing locally.*
## Does this pull request potentially affect one of the following parts
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1104528834
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +62,31 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
+ public void process(Long appId, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(appId, checkPoint));
+ }
+
+ private void process(Long appId, @Nonnull CheckPoints.CheckPoint checkPoint) {
Application application = applicationService.getById(appId);
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, application.getJobId());
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(checkPoint, latestChkId)) {
Review Comment:
> However, if the savepoint store-logic follow checkpoint, here would be some bug-case. If the triggered savepoint is completed after a completed checkpoint whose chk-id is bigger than the savepoint, the savepoint will be discard based on the current logic.
When it happened, the latestChkId is bigger than the savepoint. So `shouldStoreAsSavepoint(checkPoint, latestChkId)` returns false and the savepoint still be discarded, right?
Please correct me if I'm wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1102770619
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/controller/SavePointController.java:
##########
@@ -64,4 +66,11 @@ public RestResponse delete(Long id) throws InternalException {
Boolean deleted = savePointService.delete(id, application);
return RestResponse.success(deleted);
}
+
+ @PostMapping("trigger")
+ @RequiresPermissions("savepoint:trigger")
+ public RestResponse trigger(Long appId, @Nullable String savepointPath) {
Review Comment:
done. thank you.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -114,6 +128,41 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application app = appService.getById(appId);
+ AssertUtils.notNull(app, String.format("The application %s doesn't exist.", appId));
+ String trackingUrl = app.getJobManagerUrl();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(trackingUrl),
+ String.format("The flink trackingUrl for app[%s] isn't available.", appId));
+ AssertUtils.state(
+ StringUtils.isNotEmpty(app.getJobId()),
+ String.format("The jobId of application[%s] is absent.", appId));
+ String triggerId = triggerSavepoint(trackingUrl, app, savepointPath);
+ log.info("Request savepoint successful in triggerId {}", triggerId);
+ }
+
+ private String triggerSavepoint(
Review Comment:
updated. Thx for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1115156625
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +91,29 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
- Application application = applicationService.getById(appId);
+ public void process(Application application, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(application, checkPoint));
+ }
+
+ private void process(Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ String jobID = application.getJobId();
+ Long appId = application.getId();
CheckPointStatus status = checkPoint.getCheckPointStatus();
+ CheckPointKey checkPointKey = new CheckPointKey(appId, jobID, checkPoint.getId());
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
- checkPointCache.put(cacheId, checkPoint.getId());
+ if (shouldStoreAsSavepoint(checkPointKey, checkPoint)) {
+ savepointedCache.put(checkPointKey.getSavePointId(), DEFAULT_FLAG_BYTE);
+ saveSavepoint(checkPoint, application.getId());
Review Comment:
after saveSaveoint,would be better to call `savepointedCache.invalidate(checkPointKey.getSavePointId());`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1097553336
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -114,6 +127,41 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application app = appService.getById(appId);
+ AssertUtils.notNull(app, String.format("The application %s doesn't exist.", appId));
+ String trackingUrl = app.getJobManagerUrl();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(trackingUrl),
+ String.format("The flink trackingUrl for app[%s] isn't available.", appId));
+ String jobId = app.getJobId();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(jobId),
+ String.format("The jobId of application[%s] is absent.", appId));
+ String triggerId = triggerSavepoint(trackingUrl, jobId, savepointPath);
+ log.info("Request savepoint successful in triggerId {}", triggerId);
+ }
+
+ private String triggerSavepoint(
+ String trackingUrl, String jobId, @Nullable String savepointPath) {
+ String response = null;
+ try {
+ String url = String.format("%s/jobs/%s/savepoints", trackingUrl, jobId);
+ String postBody =
+ OBJECT_MAPPER.writeValueAsString(new SavepointTriggerRequestBody(savepointPath, false));
+ response = HttpClientUtils.httpPostRequest(url, postBody);
+ TriggerResponse triggerResponse = OBJECT_MAPPER.readValue(response, TriggerResponse.class);
+ return triggerResponse.getTriggerId().toString();
+ } catch (Exception e) {
+ throw new RuntimeException(
Review Comment:
> I see @RocMarshal changed it to `ApiException`, @wolfboys , could you please help check again? Should we use the `ApiAlertException` or `ApiException`?
`ApiAlertException `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#issuecomment-1407509022
Hi @RocMarshal , thanks for your contribution.
> @1996fanrui If you are free, can you review it first?
I'm on vacation these days, I can start review late next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#issuecomment-1407548359
> Hi @RocMarshal , thanks for your contribution.
>
>
>
> > @1996fanrui If you are free, can you review it first?
>
>
>
> I'm on vacation these days, I can start review late next week.
have a good time,I will review it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1112051673
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +66,30 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
- Application application = applicationService.getById(appId);
+ public void process(Application application, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(application, checkPoint));
+ }
+
+ private void process(Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ String jobID = application.getJobId();
+ Long appId = application.getId();
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ if (shouldStoreAsSavepoint(appId, jobID, checkPoint)) {
Review Comment:
I suggest new create a bean to store these values(appId, jobId, checkid), the readability will be better. e.g:
```
@Data
public static class CheckPointKey {
private Long appId;
private String jobId;
private Long checkId;
public CheckPointKey(Long appId, String jobId, Long checkId) {
this.appId = appId;
this.jobId = jobId;
this.checkId = checkId;
}
public String getSavePointId() {
return String.format("%s_%s_%s", appId, jobId, checkId);
}
public String getCheckPointId() {
return String.format("%s_%s", appId, jobId);
}
}
```
what you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1092781683
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -36,19 +38,30 @@
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.flink.runtime.rest.handler.async.TriggerResponse;
+import org.apache.flink.runtime.rest.messages.job.savepoints.SavepointTriggerRequestBody;
+import org.apache.flink.runtime.rest.util.RestMapperUtils;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;
+import javax.annotation.Nullable;
+
@Slf4j
@Service
@Transactional(propagation = Propagation.SUPPORTS, readOnly = true, rollbackFor = Exception.class)
public class SavePointServiceImpl extends ServiceImpl<SavePointMapper, SavePoint>
implements SavePointService {
+ private static final ObjectMapper objMapper = RestMapperUtils.getStrictObjectMapper();
+
@Autowired private FlinkEnvService flinkEnvService;
+ @Autowired private ApplicationMapper appMapper;
Review Comment:
It is not recommended to import other entity Mapper, need to change:
` @Autowired private ApplicationService applicationService;`
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/controller/ApplicationController.java:
##########
@@ -342,6 +347,12 @@ public RestResponse optionlog(ApplicationLog applicationLog, RestRequest request
return RestResponse.success(applicationList);
}
+ @PostMapping("triggerSavepoint")
Review Comment:
I recommend move this method into `SavePointController`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1094631449
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/ApplicationServiceImpl.java:
##########
@@ -1267,8 +1268,14 @@ public void cancel(Application appParam) throws Exception {
savePoint.setAppId(application.getId());
savePoint.setLatest(true);
savePoint.setType(CheckPointType.SAVEPOINT.get());
- savePoint.setTriggerTime(now);
+ savePoint.setTriggerTime(triggerTime);
savePoint.setCreateTime(now);
+ savePoint.setJobId(cancelRequest.jobId());
+ savePoint.setEndTime(now);
+ savePoint.setClusterId(application.getFlinkClusterId());
+ if (appParam.getExecutionMode() != null) {
+ savePoint.setExecutionMode(ExecutionMode.of(appParam.getExecutionMode()).name());
Review Comment:
The cxecutionMode of savepoint is in type `string`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1096672249
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/metrics/flink/CheckPoints.java:
##########
@@ -82,5 +96,20 @@ public String getPath() {
@Data
public static class Latest implements Serializable {
private CheckPoint completed;
+ private CheckPoint savepoint;
+
+ @JsonIgnore
+ public List<CheckPoint> getChkAndSavepointWithTimeAscOrder() {
+ List<CheckPoint> checkPoints = new ArrayList<>();
+ if (Objects.nonNull(completed)) {
+ checkPoints.add(completed);
+ }
+ if (Objects.nonNull(savepoint)) {
+ checkPoints.add(savepoint);
+ }
+ return checkPoints.stream()
+ .sorted(Comparator.comparingLong(chk -> chk.triggerTimestamp))
Review Comment:
How about sort the list by `checkpointId`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1096483433
##########
streampark-console/streampark-console-service/src/main/assembly/script/data/mysql-data.sql:
##########
@@ -117,6 +117,7 @@ insert into `t_menu` values (100066, 100015, 'view', null, null, 'app:view', nul
insert into `t_menu` values (100067, 100053, 'view', NULL, NULL, 'variable:view', NULL, '1', 1, null, now(), now());
insert into `t_menu` values (100068, 100033, 'view', null, null, 'setting:view', null, '1', 1, null, now(), now());
insert into `t_menu` values (100069, 100053, 'depend view', null, null, 'variable:depend_apps', null, '1', 1, NULL, now(), now());
+insert into `t_menu` values (100070, 100015, 'savepoint trigger', null, null, 'savepoint:triggerSavepoint', null, '1', 1, null, now(), now());
Review Comment:
updated.
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -210,12 +210,19 @@ drop table if exists `t_flink_savepoint`;
create table `t_flink_savepoint` (
`id` bigint not null auto_increment,
`app_id` bigint not null,
+ `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
Review Comment:
updated.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -36,19 +38,30 @@
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.flink.runtime.rest.handler.async.TriggerResponse;
+import org.apache.flink.runtime.rest.messages.job.savepoints.SavepointTriggerRequestBody;
+import org.apache.flink.runtime.rest.util.RestMapperUtils;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;
+import javax.annotation.Nullable;
+
@Slf4j
@Service
@Transactional(propagation = Propagation.SUPPORTS, readOnly = true, rollbackFor = Exception.class)
public class SavePointServiceImpl extends ServiceImpl<SavePointMapper, SavePoint>
implements SavePointService {
+ private static final ObjectMapper objMapper = RestMapperUtils.getStrictObjectMapper();
Review Comment:
updated.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -108,6 +112,23 @@ public void process(Long appId, CheckPoints checkPoints) {
}
}
+ private boolean shouldProcessFailedTrigger(
+ CheckPoints.CheckPoint checkPoint, Application application, CheckPointStatus status) {
+ return CheckPointStatus.FAILED.equals(status)
+ && !checkPoint.getIsSavepoint()
+ && application.cpFailedTrigger();
+ }
+
+ private boolean checkpointNeedStore(
+ Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ LambdaQueryWrapper<SavePoint> queryWrapper =
+ new LambdaQueryWrapper<SavePoint>()
+ .eq(SavePoint::getAppId, application.getAppId())
+ .eq(SavePoint::getJobId, application.getJobId())
Review Comment:
updated.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -114,6 +127,42 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public Boolean triggerSavepoint(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application app = appService.getById(appId);
+ AssertUtils.notNull(app, String.format("The application %s doesn't exist.", appId));
+ String trackingUrl = app.getJobManagerUrl();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(trackingUrl),
+ String.format("The flink trackingUrl for app[%s] isn't available.", appId));
+ String jobId = app.getJobId();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(jobId),
+ String.format("The jobId of application[%s] is absent.", appId));
+ String triggerId = triggerSavepoint(trackingUrl, jobId, savepointPath);
+ log.info("Request savepoint successful in triggerId {}", triggerId);
+ return true;
Review Comment:
updated.
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -210,12 +210,19 @@ drop table if exists `t_flink_savepoint`;
create table `t_flink_savepoint` (
`id` bigint not null auto_increment,
`app_id` bigint not null,
+ `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `status` VARCHAR(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'COMPLETED',
`chk_id` bigint default null,
`type` tinyint default null,
- `path` varchar(255) collate utf8mb4_general_ci default null,
+ `path` varchar(2048) collate utf8mb4_general_ci default null,
`latest` tinyint not null default 1,
`trigger_time` datetime default null,
+ `execution_mode` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
Review Comment:
removed
##########
streampark-console/streampark-console-service/src/main/assembly/script/upgrade/mysql/2.1.0.sql:
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+-- ISSUE-2192 DDL & DML Start
+
+alter table `t_flink_savepoint` modify column `path` varchar(1024) collate utf8mb4_general_ci default null;
+
+insert into `t_menu` values (100070, 100015, 'savepoint trigger', null, null, 'savepoint:trigger', null, '1', 1, null, now(), now());
Review Comment:
@1996fanrui sorry for the missing according to followed the 2.0.0.sql status. I'll add it.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/ApplicationServiceImpl.java:
##########
@@ -1267,8 +1268,14 @@ public void cancel(Application appParam) throws Exception {
savePoint.setAppId(application.getId());
savePoint.setLatest(true);
savePoint.setType(CheckPointType.SAVEPOINT.get());
- savePoint.setTriggerTime(now);
+ savePoint.setTriggerTime(triggerTime);
savePoint.setCreateTime(now);
+ savePoint.setJobId(cancelRequest.jobId());
+ savePoint.setEndTime(now);
+ savePoint.setClusterId(application.getFlinkClusterId());
+ if (appParam.getExecutionMode() != null) {
+ savePoint.setExecutionMode(ExecutionMode.of(appParam.getExecutionMode()).name());
Review Comment:
removed the field.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/ApplicationServiceImpl.java:
##########
@@ -1258,7 +1259,7 @@ public void cancel(Application appParam) throws Exception {
savePoint.setAppId(application.getId());
savePoint.setLatest(true);
savePoint.setType(CheckPointType.SAVEPOINT.get());
- savePoint.setTriggerTime(now);
Review Comment:
thx~
@1996fanrui did you mean that the `Date now = new Date()` should be inlined into `savePoint.setCreateTime(new Date())` ?
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -114,6 +127,41 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application app = appService.getById(appId);
+ AssertUtils.notNull(app, String.format("The application %s doesn't exist.", appId));
+ String trackingUrl = app.getJobManagerUrl();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(trackingUrl),
+ String.format("The flink trackingUrl for app[%s] isn't available.", appId));
+ String jobId = app.getJobId();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(jobId),
+ String.format("The jobId of application[%s] is absent.", appId));
+ String triggerId = triggerSavepoint(trackingUrl, jobId, savepointPath);
+ log.info("Request savepoint successful in triggerId {}", triggerId);
+ }
+
+ private String triggerSavepoint(
+ String trackingUrl, String jobId, @Nullable String savepointPath) {
+ String response = null;
+ try {
+ String url = String.format("%s/jobs/%s/savepoints", trackingUrl, jobId);
+ String postBody =
+ OBJECT_MAPPER.writeValueAsString(new SavepointTriggerRequestBody(savepointPath, false));
+ response = HttpClientUtils.httpPostRequest(url, postBody);
+ TriggerResponse triggerResponse = OBJECT_MAPPER.readValue(response, TriggerResponse.class);
+ return triggerResponse.getTriggerId().toString();
+ } catch (Exception e) {
+ throw new RuntimeException(
Review Comment:
@1996fanrui @wolfboys Should it be `ApiAlertException` ?
But the current coding is `AlertException` instead of `ApiAlertException`.
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -210,12 +210,19 @@ drop table if exists `t_flink_savepoint`;
create table `t_flink_savepoint` (
`id` bigint not null auto_increment,
`app_id` bigint not null,
+ `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `status` VARCHAR(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'COMPLETED',
Review Comment:
removed
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -108,6 +112,23 @@ public void process(Long appId, CheckPoints checkPoints) {
}
}
+ private boolean shouldProcessFailedTrigger(
+ CheckPoints.CheckPoint checkPoint, Application application, CheckPointStatus status) {
+ return CheckPointStatus.FAILED.equals(status)
+ && !checkPoint.getIsSavepoint()
+ && application.cpFailedTrigger();
+ }
+
+ private boolean checkpointNeedStore(
+ Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ LambdaQueryWrapper<SavePoint> queryWrapper =
Review Comment:
Thanks for the clarify.
Updated.
##########
streampark-console/streampark-console-service/src/main/assembly/script/upgrade/ISSUE-2192/schema/2192-mysql.sql:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+alter table `t_flink_savepoint` add column `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL after `app_id`;
Review Comment:
removed it into 2.1.0.sql
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -210,12 +210,19 @@ drop table if exists `t_flink_savepoint`;
create table `t_flink_savepoint` (
`id` bigint not null auto_increment,
`app_id` bigint not null,
+ `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `status` VARCHAR(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'COMPLETED',
`chk_id` bigint default null,
`type` tinyint default null,
- `path` varchar(255) collate utf8mb4_general_ci default null,
+ `path` varchar(2048) collate utf8mb4_general_ci default null,
`latest` tinyint not null default 1,
`trigger_time` datetime default null,
+ `execution_mode` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `cluster_id` bigint DEFAULT NULL,
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1096708736
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -114,6 +127,41 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application app = appService.getById(appId);
+ AssertUtils.notNull(app, String.format("The application %s doesn't exist.", appId));
+ String trackingUrl = app.getJobManagerUrl();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(trackingUrl),
+ String.format("The flink trackingUrl for app[%s] isn't available.", appId));
+ String jobId = app.getJobId();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(jobId),
+ String.format("The jobId of application[%s] is absent.", appId));
+ String triggerId = triggerSavepoint(trackingUrl, jobId, savepointPath);
+ log.info("Request savepoint successful in triggerId {}", triggerId);
+ }
+
+ private String triggerSavepoint(
+ String trackingUrl, String jobId, @Nullable String savepointPath) {
+ String response = null;
+ try {
+ String url = String.format("%s/jobs/%s/savepoints", trackingUrl, jobId);
+ String postBody =
+ OBJECT_MAPPER.writeValueAsString(new SavepointTriggerRequestBody(savepointPath, false));
+ response = HttpClientUtils.httpPostRequest(url, postBody);
+ TriggerResponse triggerResponse = OBJECT_MAPPER.readValue(response, TriggerResponse.class);
+ return triggerResponse.getTriggerId().toString();
+ } catch (Exception e) {
+ throw new RuntimeException(
Review Comment:
> Hi @wolfboys, we should use the `ApiAlertException` here, right?
Yes, we should use the `ApiAlertException`, savepoint manually triggered by the user . if an exception occurs, should be exposed to the user
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1104601687
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +62,31 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
+ public void process(Long appId, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(appId, checkPoint));
+ }
+
+ private void process(Long appId, @Nonnull CheckPoints.CheckPoint checkPoint) {
Application application = applicationService.getById(appId);
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, application.getJobId());
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(checkPoint, latestChkId)) {
+ savepointedCache.put(checkPoint.getExternalPath(), DEFAULT_FLAG_BYTE);
+ if ((latestChkId == null || checkPoint.getId() > latestChkId)) {
+ checkPointCache.put(cacheId, checkPoint.getId());
Review Comment:
If chk-id of savepoint is bigger than the latest chk-id in the checkpoint cache, the chk-id of savepoint is useful for the checkpoints store.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1108595041
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -171,6 +218,182 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public String getSavePointPath(Application appParam) throws Exception {
+ Application application = applicationService.getById(appParam.getId());
+
+ // 1) properties have the highest priority, read the properties are set: -Dstate.savepoints.dir
+ String savepointPath =
+ FlinkClient.extractDynamicPropertiesAsJava(application.getDynamicProperties())
+ .get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+
+ // Application conf configuration has the second priority. If it is a streampark|flinksql type
+ // task,
+ // see if Application conf is configured when the task is defined, if checkpoints are configured
+ // and enabled,
+ // read `state.savepoints.dir`
+ if (StringUtils.isBlank(savepointPath)) {
+ if (application.isStreamParkJob() || application.isFlinkSqlJob()) {
+ ApplicationConfig applicationConfig = configService.getEffective(application.getId());
+ if (applicationConfig != null) {
+ Map<String, String> map = applicationConfig.readConfig();
+ if (FlinkUtils.isCheckpointEnabled(map)) {
+ savepointPath = map.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+ }
+ }
+
+ // 3) If the savepoint is not obtained above, try to obtain the savepoint path according to the
+ // deployment type (remote|on yarn)
+ if (StringUtils.isBlank(savepointPath)) {
+ // 3.1) At the remote mode, request the flink webui interface to get the savepoint path
+ if (ExecutionMode.isRemoteMode(application.getExecutionMode())) {
+ FlinkCluster cluster = flinkClusterService.getById(application.getFlinkClusterId());
+ Utils.required(
+ cluster != null,
+ String.format(
+ "The clusterId=%s cannot be find, maybe the clusterId is wrong or "
+ + "the cluster has been deleted. Please contact the Admin.",
+ application.getFlinkClusterId()));
+ Map<String, String> config = cluster.getFlinkConfig();
+ if (!config.isEmpty()) {
+ savepointPath = config.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ } else {
+ // 3.2) At the yarn or k8s mode, then read the savepoint in flink-conf.yml in the bound
+ // flink
+ FlinkEnv flinkEnv = flinkEnvService.getById(application.getVersionId());
+ savepointPath =
+ flinkEnv.convertFlinkYamlAsMap().get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+
+ return savepointPath;
+ }
+
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application application = applicationService.getById(appId);
+
+ FlinkRESTAPIWatcher.addSavepoint(application.getId());
+
+ application.setOptionState(OptionState.SAVEPOINTING.getValue());
+ application.setOptionTime(new Date());
+ this.applicationService.updateById(application);
+
+ FlinkEnv flinkEnv = flinkEnvService.getById(application.getVersionId());
+
+ // infer savepoint
+ String customSavepoint = this.getFinalSavepointDir(savepointPath, application);
+
+ FlinkCluster cluster = flinkClusterService.getById(application.getFlinkClusterId());
+ String clusterId = getClusterId(application, cluster);
+
+ Map<String, Object> properties = this.tryGetRestProps(application, cluster);
+
+ TriggerSavepointRequest request =
+ new TriggerSavepointRequest(
+ flinkEnv.getFlinkVersion(),
+ ExecutionMode.of(application.getExecutionMode()),
+ clusterId,
+ application.getJobId(),
+ customSavepoint,
+ application.getK8sNamespace(),
+ properties);
+
+ CompletableFuture<SavepointResponse> savepointFuture =
+ CompletableFuture.supplyAsync(() -> FlinkClient.triggerSavepoint(request), executorService);
+
+ handleSavepointResponseFuture(application, savepointFuture);
+ }
+
+ private void handleSavepointResponseFuture(
+ Application application, CompletableFuture<SavepointResponse> savepointFuture) {
+ CompletableFutureUtils.runTimeout(
+ savepointFuture,
+ 10L,
+ TimeUnit.MINUTES,
+ savepointResponse -> {
+ if (savepointResponse != null && savepointResponse.savePointDir() != null) {
+ String savePointDir = savepointResponse.savePointDir();
+ log.info("Request savepoint successful, savepointDir: {}", savePointDir);
+ }
+ },
+ e -> {
+ log.error("Trigger savepoint for flink job failed.", e);
+ ApplicationLog log = new ApplicationLog();
+ log.setAppId(application.getId());
+ log.setYarnAppId(application.getClusterId());
+ log.setOptionTime(new Date());
+ String exception = Utils.stringifyException(e);
+ log.setException(exception);
+ if (!(e instanceof TimeoutException)) {
+ log.setSuccess(false);
+ }
+ applicationLogService.save(log);
+ })
+ .whenComplete(
+ (t, e) -> {
+ application.setOptionState(OptionState.NONE.getValue());
+ application.setOptionTime(new Date());
+ applicationService.update(application);
Review Comment:
Get it. And I'll reorganize this mechanism about `FlinkRESTAPIWatcher` after the MR.
Thanks for your reminding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1111225490
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +66,32 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
- Application application = applicationService.getById(appId);
+ public void process(Application application, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(application, checkPoint));
+ }
+
+ private void process(Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ String jobID = application.getJobId();
+ Long appId = application.getId();
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, jobID);
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(appId, jobID, checkPoint)) {
+ savepointedCache.put(
+ getCacheIdForSavepoint(appId, jobID, checkPoint.getId()), DEFAULT_FLAG_BYTE);
+ saveSavepoint(
+ checkPoint,
+ application.getId(),
+ latestChkId == null || checkPoint.getId() > latestChkId);
+ return;
+ }
+ if (shouldStoreAsCheckpoint(checkPoint, latestChkId)) {
Review Comment:
We need consider whether the savepoint should be stored as the latest.
I check the `CheckpointProcessor#saveSavepoint` method and maybe there would be a bug on `latest` value when the chk-n checkpoint completed before chk-n-1 savepoint.
Please let me know your opinion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1096668226
##########
streampark-console/streampark-console-service/src/main/assembly/script/upgrade/mysql/2.1.0.sql:
##########
@@ -0,0 +1,24 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+-- ISSUE-2192 DDL & DML Start
+
+alter table `t_flink_savepoint` modify column `path` varchar(1024) collate utf8mb4_general_ci default null;
+
+insert into `t_menu` values (100070, 100015, 'savepoint trigger', null, null, 'savepoint:trigger', null, '1', 1, null, now(), now());
Review Comment:
The upgrade sql of pgsql is missed.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -114,6 +127,41 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application app = appService.getById(appId);
+ AssertUtils.notNull(app, String.format("The application %s doesn't exist.", appId));
+ String trackingUrl = app.getJobManagerUrl();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(trackingUrl),
+ String.format("The flink trackingUrl for app[%s] isn't available.", appId));
+ String jobId = app.getJobId();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(jobId),
+ String.format("The jobId of application[%s] is absent.", appId));
+ String triggerId = triggerSavepoint(trackingUrl, jobId, savepointPath);
+ log.info("Request savepoint successful in triggerId {}", triggerId);
+ }
+
+ private String triggerSavepoint(
+ String trackingUrl, String jobId, @Nullable String savepointPath) {
+ String response = null;
+ try {
+ String url = String.format("%s/jobs/%s/savepoints", trackingUrl, jobId);
+ String postBody =
+ OBJECT_MAPPER.writeValueAsString(new SavepointTriggerRequestBody(savepointPath, false));
+ response = HttpClientUtils.httpPostRequest(url, postBody);
+ TriggerResponse triggerResponse = OBJECT_MAPPER.readValue(response, TriggerResponse.class);
+ return triggerResponse.getTriggerId().toString();
+ } catch (Exception e) {
+ throw new RuntimeException(
Review Comment:
Hi @wolfboys, we should use the `ApiAlertException` here, right?
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/ApplicationServiceImpl.java:
##########
@@ -1258,7 +1259,7 @@ public void cancel(Application appParam) throws Exception {
savePoint.setAppId(application.getId());
savePoint.setLatest(true);
savePoint.setType(CheckPointType.SAVEPOINT.get());
- savePoint.setTriggerTime(now);
Review Comment:
Could the `now` be removed?
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/FlinkRESTAPIWatcher.java:
##########
@@ -357,7 +357,9 @@ private void handleCheckPoints(Application application) throws Exception {
FlinkCluster flinkCluster = getFlinkCluster(application);
CheckPoints checkPoints = httpCheckpoints(application, flinkCluster);
if (checkPoints != null) {
- checkpointProcessor.process(application.getId(), checkPoints);
+ checkPoints
+ .getLatestChkAndSavepointWithTimeAscOrder()
Review Comment:
I prefer move the logic into `checkpointProcessor`, this class is `FlinkRESTAPIWatcher`, it shouldn't care about these logics.
Overall, don't change this file, and we still pass the `CheckPoints checkPoints` to `checkpointProcessor.process(application.getId(), checkPoints);`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1103476248
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/metrics/flink/CheckPoints.java:
##########
@@ -82,5 +96,20 @@ public String getPath() {
@Data
public static class Latest implements Serializable {
private CheckPoint completed;
+ private CheckPoint savepoint;
+
+ @JsonIgnore
+ public List<CheckPoint> getLatestChkSavepointByIdOrder() {
Review Comment:
Okay~.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#issuecomment-1416658960
> > @1996fanrui Thx so much for the comments. I'll update it after the comments discussed.
>
> Hi @RocMarshal , thanks for your quick feedback.
>
> From your feedback, I found two main problems:
>
> * You added a lot of code that is not currently needed in this PR, such as: status, end_time, execution_mode, cluster_id, etc. And we can add them if they're needed in the future.
> * Modify the logic of `CheckpointProcessor` as little as possible. The code of the master branch determines whether checkpoint/savepoint needs to be stored in the database according to the cache, and I think it is enough for now. Your code will access to the database frequently, it isn't friendly to database.
>
> Overall, this PR should only contain logic to trigger savepoint and fetch savepoint list.
@1996fanrui thx for the response, and I'll keep the `minimum-change` for the next update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] lvshaokang commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1098114295
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/controller/SavePointController.java:
##########
@@ -64,4 +66,11 @@ public RestResponse delete(Long id) throws InternalException {
Boolean deleted = savePointService.delete(id, application);
return RestResponse.success(deleted);
}
+
+ @PostMapping("trigger")
+ @RequiresPermissions("savepoint:trigger")
+ public RestResponse trigger(Long appId, @Nullable String savepointPath) {
Review Comment:
Could you add swagger annotations to help third-party services interface with streampark?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1109821875
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/metrics/flink/CheckPoints.java:
##########
@@ -82,5 +96,20 @@ public String getPath() {
@Data
public static class Latest implements Serializable {
private CheckPoint completed;
+ private CheckPoint savepoint;
+
+ @JsonIgnore
+ public List<CheckPoint> getLatestCheckpoint() {
+ List<CheckPoint> checkPoints = new ArrayList<>();
+ if (Objects.nonNull(completed)) {
+ checkPoints.add(completed);
+ }
+ if (Objects.nonNull(savepoint)) {
+ checkPoints.add(savepoint);
+ }
+ return checkPoints.stream()
+ .sorted(Comparator.comparingLong(chk -> chk.id))
Review Comment:
The sort isn't necessary, right?
If so, we can return the checkpoints directly.
##########
streampark-flink/streampark-flink-client/streampark-flink-client-core/src/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeSessionSubmit.scala:
##########
@@ -212,4 +212,9 @@ object KubernetesNativeSessionSubmit extends KubernetesNativeSubmitTrait with Lo
Utils.close(kubeClient)
}
}
+
+ override def doTriggerSavepoint(request: TriggerSavepointRequest, flinkConfig: Configuration): SavepointResponse = {
+ flinkConfig.safeSet(DeploymentOptions.TARGET, ExecutionMode.KUBERNETES_NATIVE_APPLICATION.getName)
Review Comment:
```suggestion
flinkConfig.safeSet(DeploymentOptions.TARGET, ExecutionMode. KUBERNETES_NATIVE_SESSION.getName)
```
Is it a bug?
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -54,6 +85,24 @@ public class SavePointServiceImpl extends ServiceImpl<SavePointMapper, SavePoint
@Autowired private ApplicationService applicationService;
+ @Autowired private ApplicationConfigService configService;
+
+ @Autowired private FlinkClusterService flinkClusterService;
+
+ @Autowired private ApplicationLogService applicationLogService;
+
+ @Autowired private FlinkRESTAPIWatcher flinkRESTAPIWatcher;
+
+ private final ExecutorService executorService =
+ new ThreadPoolExecutor(
+ Runtime.getRuntime().availableProcessors() * 5,
+ Runtime.getRuntime().availableProcessors() * 10,
+ 60L,
+ TimeUnit.SECONDS,
+ new LinkedBlockingQueue<>(1024),
+ ThreadUtils.threadFactory("streampark-trigger-savepoint-executor"),
Review Comment:
```suggestion
ThreadUtils.threadFactory("trigger-savepoint-executor"),
```
`streampark` isn't needed due to all `thread pool` belong to StreamPark 😂
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +66,29 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
+ public void process(Long appId, String jobID, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(appId, jobID, checkPoint));
+ }
+
+ private void process(Long appId, String jobID, @Nonnull CheckPoints.CheckPoint checkPoint) {
Application application = applicationService.getById(appId);
Review Comment:
This query can be removed, and we can pass some parameters from caller. It can reduce the pressure of DB.
For example, jobID, application.cpFailedTrigger(), etc.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +66,29 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
+ public void process(Long appId, String jobID, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(appId, jobID, checkPoint));
+ }
+
+ private void process(Long appId, String jobID, @Nonnull CheckPoints.CheckPoint checkPoint) {
Application application = applicationService.getById(appId);
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, application.getJobId());
Review Comment:
```suggestion
String cacheId = getCacheIdForCheckpoint(appId, jobID);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1107350361
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -41,9 +44,16 @@
@Component
public class CheckpointProcessor {
+ private static final Byte DEFAULT_FLAG_BYTE = Byte.valueOf("0");
+ private static final Integer SAVEPOINT_CACHE_HOUR = 1;
+
private final Cache<String, Long> checkPointCache =
Caffeine.newBuilder().expireAfterAccess(1, TimeUnit.DAYS).build();
+ /** Cache to store the savepoint path if be stored in the db. */
+ private final Cache<String, Byte> savepointedCache =
Review Comment:
Using the savepoint path as the cache key, because you want to save the older checkpoint, right?
If so, it's better to add some comments here to help other developers to understand this logic.
Also, I'm not sure whether the `app_ip + job_id + chk_id` can be the cache key ? If it can work, I think it is better than savepoint path.
Because we want to save every savepoint and the latest check point , so the savepoint cache key added the chkid.
WDYT?
And sorry for the late response, you know, I'm very busy this month.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#issuecomment-1407406301
Thanks for your contribution, I will review later, @1996fanrui If you are free, can you review it first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1105144165
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +62,31 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
+ public void process(Long appId, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(appId, checkPoint));
+ }
+
+ private void process(Long appId, @Nonnull CheckPoints.CheckPoint checkPoint) {
Application application = applicationService.getById(appId);
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, application.getJobId());
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(checkPoint, latestChkId)) {
+ savepointedCache.put(checkPoint.getExternalPath(), DEFAULT_FLAG_BYTE);
Review Comment:
> We can't judge the savepoint if be stored out of TTL.
Why can't? Doesn't it have TriggerTime? If triggerTime < current time - TTL, it should be discarded, right?
> I‘m hesitating whether we should take savepoint as more important than checkpoint.
I think it's more important due to it's triggered maunally. User might want to use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1111215293
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +66,32 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
- Application application = applicationService.getById(appId);
+ public void process(Application application, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(application, checkPoint));
+ }
+
+ private void process(Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ String jobID = application.getJobId();
+ Long appId = application.getId();
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, jobID);
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(appId, jobID, checkPoint)) {
+ savepointedCache.put(
+ getCacheIdForSavepoint(appId, jobID, checkPoint.getId()), DEFAULT_FLAG_BYTE);
+ saveSavepoint(
+ checkPoint,
+ application.getId(),
+ latestChkId == null || checkPoint.getId() > latestChkId);
+ return;
+ }
+ if (shouldStoreAsCheckpoint(checkPoint, latestChkId)) {
Review Comment:
We still use `latestChkId ` which references the `cacheId`, right?
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +66,32 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
- Application application = applicationService.getById(appId);
+ public void process(Application application, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(application, checkPoint));
+ }
+
+ private void process(Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ String jobID = application.getJobId();
+ Long appId = application.getId();
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, jobID);
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(appId, jobID, checkPoint)) {
+ savepointedCache.put(
+ getCacheIdForSavepoint(appId, jobID, checkPoint.getId()), DEFAULT_FLAG_BYTE);
+ saveSavepoint(
+ checkPoint,
+ application.getId(),
+ latestChkId == null || checkPoint.getId() > latestChkId);
+ return;
+ }
+ if (shouldStoreAsCheckpoint(checkPoint, latestChkId)) {
Review Comment:
@1996fanrui We still use `latestChkId ` which references the `cacheId`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1111250539
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +66,32 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
- Application application = applicationService.getById(appId);
+ public void process(Application application, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(application, checkPoint));
+ }
+
+ private void process(Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ String jobID = application.getJobId();
+ Long appId = application.getId();
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, jobID);
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(appId, jobID, checkPoint)) {
+ savepointedCache.put(
+ getCacheIdForSavepoint(appId, jobID, checkPoint.getId()), DEFAULT_FLAG_BYTE);
+ saveSavepoint(
+ checkPoint,
+ application.getId(),
+ latestChkId == null || checkPoint.getId() > latestChkId);
+ return;
+ }
+ if (shouldStoreAsCheckpoint(checkPoint, latestChkId)) {
Review Comment:
I don't understand why savepoint needs the latestChkId.
Actually, I think we don't need the latest field in the database. And the latest can be fill in during qurey checkpoint list. The checkpoint with max triggerTime should be the latest checkpoint.
So I prefer keep old logic that latest is always true during save chk or savepoint.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1111255615
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +66,32 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
- Application application = applicationService.getById(appId);
+ public void process(Application application, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(application, checkPoint));
+ }
+
+ private void process(Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ String jobID = application.getJobId();
+ Long appId = application.getId();
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, jobID);
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(appId, jobID, checkPoint)) {
+ savepointedCache.put(
+ getCacheIdForSavepoint(appId, jobID, checkPoint.getId()), DEFAULT_FLAG_BYTE);
+ saveSavepoint(
+ checkPoint,
+ application.getId(),
+ latestChkId == null || checkPoint.getId() > latestChkId);
+ return;
+ }
+ if (shouldStoreAsCheckpoint(checkPoint, latestChkId)) {
Review Comment:
By the way, the cache still have bug when streampark restarts.
Because the cache will be empty after restart. So I prefer do less here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1093270570
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -36,19 +38,30 @@
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.flink.runtime.rest.handler.async.TriggerResponse;
+import org.apache.flink.runtime.rest.messages.job.savepoints.SavepointTriggerRequestBody;
+import org.apache.flink.runtime.rest.util.RestMapperUtils;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;
+import javax.annotation.Nullable;
+
@Slf4j
@Service
@Transactional(propagation = Propagation.SUPPORTS, readOnly = true, rollbackFor = Exception.class)
public class SavePointServiceImpl extends ServiceImpl<SavePointMapper, SavePoint>
implements SavePointService {
+ private static final ObjectMapper objMapper = RestMapperUtils.getStrictObjectMapper();
+
@Autowired private FlinkEnvService flinkEnvService;
+ @Autowired private ApplicationMapper appMapper;
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1103617737
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -17,38 +17,100 @@
package org.apache.streampark.console.core.service.impl;
+import org.apache.streampark.common.enums.ExecutionMode;
import org.apache.streampark.common.util.AssertUtils;
+import org.apache.streampark.common.util.CompletableFutureUtils;
+import org.apache.streampark.common.util.ExceptionUtils;
+import org.apache.streampark.common.util.FlinkUtils;
+import org.apache.streampark.common.util.ThreadUtils;
import org.apache.streampark.console.base.domain.Constant;
import org.apache.streampark.console.base.domain.RestRequest;
+import org.apache.streampark.console.base.exception.ApiAlertException;
import org.apache.streampark.console.base.exception.InternalException;
import org.apache.streampark.console.base.mybatis.pager.MybatisPager;
import org.apache.streampark.console.base.util.CommonUtils;
import org.apache.streampark.console.core.entity.Application;
+import org.apache.streampark.console.core.entity.ApplicationConfig;
+import org.apache.streampark.console.core.entity.ApplicationLog;
+import org.apache.streampark.console.core.entity.FlinkCluster;
import org.apache.streampark.console.core.entity.FlinkEnv;
import org.apache.streampark.console.core.entity.SavePoint;
+import org.apache.streampark.console.core.enums.CheckPointStatus;
import org.apache.streampark.console.core.enums.CheckPointType;
+import org.apache.streampark.console.core.enums.OptionState;
import org.apache.streampark.console.core.mapper.SavePointMapper;
+import org.apache.streampark.console.core.metrics.flink.CheckPoints;
+import org.apache.streampark.console.core.service.ApplicationConfigService;
+import org.apache.streampark.console.core.service.ApplicationLogService;
+import org.apache.streampark.console.core.service.ApplicationService;
+import org.apache.streampark.console.core.service.FlinkClusterService;
import org.apache.streampark.console.core.service.FlinkEnvService;
import org.apache.streampark.console.core.service.SavePointService;
+import org.apache.streampark.console.core.task.CheckpointProcessor;
+import org.apache.streampark.console.core.task.FlinkRESTAPIWatcher;
+import org.apache.streampark.flink.client.FlinkClient;
+import org.apache.streampark.flink.client.bean.SavepointResponse;
+import org.apache.streampark.flink.client.bean.TriggerSavepointRequest;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.metadata.IPage;
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.runtime.rest.util.RestMapperUtils;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+import org.jetbrains.annotations.NotNull;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;
+import javax.annotation.Nullable;
+
+import java.net.URI;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
@Slf4j
@Service
@Transactional(propagation = Propagation.SUPPORTS, readOnly = true, rollbackFor = Exception.class)
public class SavePointServiceImpl extends ServiceImpl<SavePointMapper, SavePoint>
implements SavePointService {
+ private static final ObjectMapper OBJECT_MAPPER = RestMapperUtils.getStrictObjectMapper();
+
@Autowired private FlinkEnvService flinkEnvService;
+ @Autowired private ApplicationService appService;
+
+ @Autowired private SavePointService savePointService;
+
+ @Autowired private ApplicationConfigService configService;
+ @Autowired private FlinkClusterService flinkClusterService;
Review Comment:
Not sure if this format can be strictly constrained by `checkstyle` or `spotless`? Otherwise, it's too hard for reviewers and developers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1103634516
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -41,9 +44,16 @@
@Component
public class CheckpointProcessor {
+ private static final Byte DEFAULT_FLAG_BYTE = Byte.valueOf("0");
+ private static final Integer SAVEPOINT_CACHE_HOUR = 1;
+
private final Cache<String, Long> checkPointCache =
Caffeine.newBuilder().expireAfterAccess(1, TimeUnit.DAYS).build();
+ /** Cache to store the savepoint path if be stored in the db. */
+ private final Cache<String, Byte> savepointedCache =
Review Comment:
Thanks for your quick clarification.
> However, if the savepoint store-logic follow checkpoint, here would be some bug-case. If the triggered savepoint is completed after a completed checkpoint whose chk-id is bigger than the savepoint, the savepoint will be discard based on the current logic.
As I understand, you have ordered the checkpoint by chk-id, so the bug cannot be happened when `execution.checkpointing.max-concurrent-checkpoints == 1` due to the `checkpoint/savepoint` with a smaller `chk-id` must be completed earlier than the checkpoint with a larger `chk-id`, right?
Are you worried about `execution.checkpointing.max-concurrent-checkpoints > 1`?
And I mean the code is a bit complicated right now, not sure if it's necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1094643781
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -108,6 +112,23 @@ public void process(Long appId, CheckPoints checkPoints) {
}
}
+ private boolean shouldProcessFailedTrigger(
+ CheckPoints.CheckPoint checkPoint, Application application, CheckPointStatus status) {
+ return CheckPointStatus.FAILED.equals(status)
+ && !checkPoint.getIsSavepoint()
+ && application.cpFailedTrigger();
+ }
+
+ private boolean checkpointNeedStore(
+ Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ LambdaQueryWrapper<SavePoint> queryWrapper =
Review Comment:
A minor confusion for me form the comments.
Does the minor refactor cause the logic loss or the original implementation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1094601995
##########
streampark-console/streampark-console-service/src/main/assembly/script/data/mysql-data.sql:
##########
@@ -117,6 +117,7 @@ insert into `t_menu` values (100066, 100015, 'view', null, null, 'app:view', nul
insert into `t_menu` values (100067, 100053, 'view', NULL, NULL, 'variable:view', NULL, '1', 1, null, now(), now());
insert into `t_menu` values (100068, 100033, 'view', null, null, 'setting:view', null, '1', 1, null, now(), now());
insert into `t_menu` values (100069, 100053, 'depend view', null, null, 'variable:depend_apps', null, '1', 1, NULL, now(), now());
+insert into `t_menu` values (100070, 100015, 'savepoint trigger', null, null, 'savepoint:triggerSavepoint', null, '1', 1, null, now(), now());
Review Comment:
concise
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1103630736
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -41,9 +44,16 @@
@Component
public class CheckpointProcessor {
+ private static final Byte DEFAULT_FLAG_BYTE = Byte.valueOf("0");
+ private static final Integer SAVEPOINT_CACHE_HOUR = 1;
+
private final Cache<String, Long> checkPointCache =
Caffeine.newBuilder().expireAfterAccess(1, TimeUnit.DAYS).build();
+ /** Cache to store the savepoint path if be stored in the db. */
+ private final Cache<String, Byte> savepointedCache =
Review Comment:
1. I moved the store logic for savepoint here in order to uniform data caliber, which could supply the savepoint chk-id field.
The chk-id is meaningful to adapt the store logic of checkpoints.
2. For your question: We prefer to store all the completed savepoints and checkpoints. However, if the savepoint store-logic follow `checkpoint`, here would be some bug-case. If the triggered savepoint is completed after a completed checkpoint whose chk-id is bigger than the savepoint, the savepoint will be discard based on the current logic. So, the `savepointedCache` was introduced, If the chk-id of the completed savepoint is smaller than the latest stored checkpoint-id, we just store the savepoint whose triggertime within `savepointCache` .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#issuecomment-1433108947
hi RocMarshal please resolve conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1109911711
##########
streampark-flink/streampark-flink-client/streampark-flink-client-core/src/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeSessionSubmit.scala:
##########
@@ -212,4 +212,9 @@ object KubernetesNativeSessionSubmit extends KubernetesNativeSubmitTrait with Lo
Utils.close(kubeClient)
}
}
+
+ override def doTriggerSavepoint(request: TriggerSavepointRequest, flinkConfig: Configuration): SavepointResponse = {
+ flinkConfig.safeSet(DeploymentOptions.TARGET, ExecutionMode.KUBERNETES_NATIVE_APPLICATION.getName)
Review Comment:
The lines is coded follow the original lines in the class.
@wolfboys could you help check it ? I'm not familiar with the k8s execution mode. As mentioned above comments, I test for the feature on all execution mode besides k8s mode.
CC @1996fanrui
thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1108591533
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -171,6 +218,182 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public String getSavePointPath(Application appParam) throws Exception {
+ Application application = applicationService.getById(appParam.getId());
+
+ // 1) properties have the highest priority, read the properties are set: -Dstate.savepoints.dir
+ String savepointPath =
+ FlinkClient.extractDynamicPropertiesAsJava(application.getDynamicProperties())
+ .get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+
+ // Application conf configuration has the second priority. If it is a streampark|flinksql type
+ // task,
+ // see if Application conf is configured when the task is defined, if checkpoints are configured
+ // and enabled,
+ // read `state.savepoints.dir`
+ if (StringUtils.isBlank(savepointPath)) {
+ if (application.isStreamParkJob() || application.isFlinkSqlJob()) {
+ ApplicationConfig applicationConfig = configService.getEffective(application.getId());
+ if (applicationConfig != null) {
+ Map<String, String> map = applicationConfig.readConfig();
+ if (FlinkUtils.isCheckpointEnabled(map)) {
+ savepointPath = map.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+ }
+ }
+
+ // 3) If the savepoint is not obtained above, try to obtain the savepoint path according to the
+ // deployment type (remote|on yarn)
+ if (StringUtils.isBlank(savepointPath)) {
+ // 3.1) At the remote mode, request the flink webui interface to get the savepoint path
+ if (ExecutionMode.isRemoteMode(application.getExecutionMode())) {
+ FlinkCluster cluster = flinkClusterService.getById(application.getFlinkClusterId());
+ Utils.required(
+ cluster != null,
+ String.format(
+ "The clusterId=%s cannot be find, maybe the clusterId is wrong or "
+ + "the cluster has been deleted. Please contact the Admin.",
+ application.getFlinkClusterId()));
+ Map<String, String> config = cluster.getFlinkConfig();
+ if (!config.isEmpty()) {
+ savepointPath = config.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ } else {
+ // 3.2) At the yarn or k8s mode, then read the savepoint in flink-conf.yml in the bound
+ // flink
+ FlinkEnv flinkEnv = flinkEnvService.getById(application.getVersionId());
+ savepointPath =
+ flinkEnv.convertFlinkYamlAsMap().get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+
+ return savepointPath;
+ }
+
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application application = applicationService.getById(appId);
+
+ FlinkRESTAPIWatcher.addSavepoint(application.getId());
+
+ application.setOptionState(OptionState.SAVEPOINTING.getValue());
+ application.setOptionTime(new Date());
+ this.applicationService.updateById(application);
+
+ FlinkEnv flinkEnv = flinkEnvService.getById(application.getVersionId());
+
+ // infer savepoint
+ String customSavepoint = this.getFinalSavepointDir(savepointPath, application);
+
+ FlinkCluster cluster = flinkClusterService.getById(application.getFlinkClusterId());
+ String clusterId = getClusterId(application, cluster);
+
+ Map<String, Object> properties = this.tryGetRestProps(application, cluster);
+
+ TriggerSavepointRequest request =
+ new TriggerSavepointRequest(
+ flinkEnv.getFlinkVersion(),
+ ExecutionMode.of(application.getExecutionMode()),
+ clusterId,
+ application.getJobId(),
+ customSavepoint,
+ application.getK8sNamespace(),
+ properties);
+
+ CompletableFuture<SavepointResponse> savepointFuture =
+ CompletableFuture.supplyAsync(() -> FlinkClient.triggerSavepoint(request), executorService);
+
+ handleSavepointResponseFuture(application, savepointFuture);
+ }
+
+ private void handleSavepointResponseFuture(
+ Application application, CompletableFuture<SavepointResponse> savepointFuture) {
+ CompletableFutureUtils.runTimeout(
+ savepointFuture,
+ 10L,
+ TimeUnit.MINUTES,
+ savepointResponse -> {
+ if (savepointResponse != null && savepointResponse.savePointDir() != null) {
+ String savePointDir = savepointResponse.savePointDir();
+ log.info("Request savepoint successful, savepointDir: {}", savePointDir);
+ }
+ },
+ e -> {
+ log.error("Trigger savepoint for flink job failed.", e);
+ ApplicationLog log = new ApplicationLog();
+ log.setAppId(application.getId());
+ log.setYarnAppId(application.getClusterId());
+ log.setOptionTime(new Date());
+ String exception = Utils.stringifyException(e);
+ log.setException(exception);
+ if (!(e instanceof TimeoutException)) {
+ log.setSuccess(false);
+ }
+ applicationLogService.save(log);
+ })
+ .whenComplete(
+ (t, e) -> {
+ application.setOptionState(OptionState.NONE.getValue());
+ application.setOptionTime(new Date());
+ applicationService.update(application);
+ });
+ }
+
+ private String getFinalSavepointDir(@Nullable String savepointPath, Application application) {
+ String result = savepointPath;
+ if (StringUtils.isEmpty(savepointPath)) {
+ try {
+ result = this.getSavePointPath(application);
+ } catch (Exception e) {
+ throw new ApiAlertException(
+ "Error in getting savepoint path for triggering savepoint for app "
+ + application.getId(),
+ e);
+ }
+ }
+ return result;
+ }
+
+ @NotNull
+ private Map<String, Object> tryGetRestProps(Application application, FlinkCluster cluster) {
+ Map<String, Object> properties = new HashMap<>();
+
+ if (ExecutionMode.isRemoteMode(application.getExecutionModeEnum())) {
+ Utils.required(
+ cluster != null,
+ String.format(
+ "The clusterId=%s cannot be find, maybe the clusterId is wrong or "
+ + "the cluster has been deleted. Please contact the Admin.",
+ application.getFlinkClusterId()));
+ URI activeAddress = cluster.getRemoteURI();
+ properties.put(RestOptions.ADDRESS.key(), activeAddress.getHost());
+ properties.put(RestOptions.PORT.key(), activeAddress.getPort());
+ }
+ return properties;
+ }
+
+ private String getClusterId(Application application, FlinkCluster cluster) {
+ if (ExecutionMode.isKubernetesMode(application.getExecutionMode())) {
+ return application.getClusterId();
+ } else if (ExecutionMode.isYarnMode(application.getExecutionMode())) {
+ if (ExecutionMode.YARN_SESSION.equals(application.getExecutionModeEnum())) {
+ Utils.required(
Review Comment:
sorry for the lines.
Because I just check the feature is healthy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1111462111
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +66,32 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
- Application application = applicationService.getById(appId);
+ public void process(Application application, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(application, checkPoint));
+ }
+
+ private void process(Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ String jobID = application.getJobId();
+ Long appId = application.getId();
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, jobID);
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(appId, jobID, checkPoint)) {
+ savepointedCache.put(
+ getCacheIdForSavepoint(appId, jobID, checkPoint.getId()), DEFAULT_FLAG_BYTE);
+ saveSavepoint(
+ checkPoint,
+ application.getId(),
+ latestChkId == null || checkPoint.getId() > latestChkId);
+ return;
+ }
+ if (shouldStoreAsCheckpoint(checkPoint, latestChkId)) {
Review Comment:
Thank you so much for the clarify. I got it & updated it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1111222699
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -54,6 +85,24 @@ public class SavePointServiceImpl extends ServiceImpl<SavePointMapper, SavePoint
@Autowired private ApplicationService applicationService;
+ @Autowired private ApplicationConfigService configService;
+
+ @Autowired private FlinkClusterService flinkClusterService;
+
+ @Autowired private ApplicationLogService applicationLogService;
+
+ @Autowired private FlinkRESTAPIWatcher flinkRESTAPIWatcher;
+
+ private final ExecutorService executorService =
+ new ThreadPoolExecutor(
+ Runtime.getRuntime().availableProcessors() * 5,
+ Runtime.getRuntime().availableProcessors() * 10,
+ 60L,
+ TimeUnit.SECONDS,
+ new LinkedBlockingQueue<>(1024),
+ ThreadUtils.threadFactory("streampark-trigger-savepoint-executor"),
Review Comment:
Could you address all comments? And please let me know if some comments don't make sense, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#issuecomment-1433219802
Thank you @wolfboys @1996fanrui very much for your review.
I updated it .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1098063884
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -114,6 +128,41 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application app = appService.getById(appId);
+ AssertUtils.notNull(app, String.format("The application %s doesn't exist.", appId));
+ String trackingUrl = app.getJobManagerUrl();
Review Comment:
only on yarn mode, `jobManagerUrl` has value, else not.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -114,6 +128,41 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application app = appService.getById(appId);
+ AssertUtils.notNull(app, String.format("The application %s doesn't exist.", appId));
+ String trackingUrl = app.getJobManagerUrl();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(trackingUrl),
+ String.format("The flink trackingUrl for app[%s] isn't available.", appId));
+ AssertUtils.state(
+ StringUtils.isNotEmpty(app.getJobId()),
+ String.format("The jobId of application[%s] is absent.", appId));
+ String triggerId = triggerSavepoint(trackingUrl, app, savepointPath);
+ log.info("Request savepoint successful in triggerId {}", triggerId);
+ }
+
+ private String triggerSavepoint(
Review Comment:
I suggest move this method in `FlinkSubmitter`, and the service layer of the platform just call it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1103649017
##########
streampark-flink/streampark-flink-client/streampark-flink-client-core/src/main/scala/org/apache/streampark/flink/client/FlinkClientHandler.scala:
##########
@@ -36,6 +36,19 @@ object FlinkClientHandler {
}
}
+ def triggerSavepoint(savepointRequest: TriggerSavepointRequest): SavepointResponse = {
+ savepointRequest.executionMode match {
+ case ExecutionMode.LOCAL => LocalSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.REMOTE => RemoteSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.YARN_APPLICATION => YarnApplicationSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.YARN_SESSION => YarnSessionSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.YARN_PER_JOB | ExecutionMode.YARN_SESSION => YarnPerJobSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.KUBERNETES_NATIVE_SESSION => KubernetesNativeSessionSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.KUBERNETES_NATIVE_APPLICATION => KubernetesNativeApplicationSubmit.triggerSavepoint(savepointRequest)
+ case _ => throw new UnsupportedOperationException(s"Unsupported ${savepointRequest.executionMode} Submit ")
+ }
+ }
+
Review Comment:
> Hi @RocMarshal , `FlinkClientHandler` is a bad case for code design, it has too many `match ... case `.
>
> It can be refactored after this PR, would you like to work on it?
>
> cc @wolfboys
Indeed, this part of the code needs to be refactored, which can be done in the next PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1103619008
##########
streampark-flink/streampark-flink-client/streampark-flink-client-core/src/main/scala/org/apache/streampark/flink/client/FlinkClientHandler.scala:
##########
@@ -36,6 +36,19 @@ object FlinkClientHandler {
}
}
+ def triggerSavepoint(savepointRequest: TriggerSavepointRequest): SavepointResponse = {
+ savepointRequest.executionMode match {
+ case ExecutionMode.LOCAL => LocalSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.REMOTE => RemoteSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.YARN_APPLICATION => YarnApplicationSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.YARN_SESSION => YarnSessionSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.YARN_PER_JOB | ExecutionMode.YARN_SESSION => YarnPerJobSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.KUBERNETES_NATIVE_SESSION => KubernetesNativeSessionSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.KUBERNETES_NATIVE_APPLICATION => KubernetesNativeApplicationSubmit.triggerSavepoint(savepointRequest)
+ case _ => throw new UnsupportedOperationException(s"Unsupported ${savepointRequest.executionMode} Submit ")
+ }
+ }
+
Review Comment:
Hi @RocMarshal , `FlinkClientHandler` is a bad case for code design, it has too many `match ... case `.
It can be refactored after this PR, would you like to work on it?
cc @wolfboys
##########
streampark-flink/streampark-flink-client/streampark-flink-client-api/src/main/scala/org/apache/streampark/flink/client/bean/SavepointRequestTrait.scala:
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.streampark.flink.client.bean
+
+import java.util.{Map => JavaMap}
+
+import org.apache.streampark.common.conf.{FlinkVersion, K8sFlinkConfig}
+import org.apache.streampark.common.enums.ExecutionMode
+
+import javax.annotation.Nullable
+
+trait SavepointRequestTrait {
Review Comment:
Should some common fields be defined inside the trait instead of sub-class? such as: flink version, jobId, etc.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -41,9 +44,16 @@
@Component
public class CheckpointProcessor {
+ private static final Byte DEFAULT_FLAG_BYTE = Byte.valueOf("0");
+ private static final Integer SAVEPOINT_CACHE_HOUR = 1;
+
private final Cache<String, Long> checkPointCache =
Caffeine.newBuilder().expireAfterAccess(1, TimeUnit.DAYS).build();
+ /** Cache to store the savepoint path if be stored in the db. */
+ private final Cache<String, Byte> savepointedCache =
Review Comment:
Why need the `savepointedCache`? Why `checkPointCache` isn't enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1103433809
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -17,38 +17,100 @@
package org.apache.streampark.console.core.service.impl;
+import org.apache.streampark.common.enums.ExecutionMode;
import org.apache.streampark.common.util.AssertUtils;
+import org.apache.streampark.common.util.CompletableFutureUtils;
+import org.apache.streampark.common.util.ExceptionUtils;
+import org.apache.streampark.common.util.FlinkUtils;
+import org.apache.streampark.common.util.ThreadUtils;
import org.apache.streampark.console.base.domain.Constant;
import org.apache.streampark.console.base.domain.RestRequest;
+import org.apache.streampark.console.base.exception.ApiAlertException;
import org.apache.streampark.console.base.exception.InternalException;
import org.apache.streampark.console.base.mybatis.pager.MybatisPager;
import org.apache.streampark.console.base.util.CommonUtils;
import org.apache.streampark.console.core.entity.Application;
+import org.apache.streampark.console.core.entity.ApplicationConfig;
+import org.apache.streampark.console.core.entity.ApplicationLog;
+import org.apache.streampark.console.core.entity.FlinkCluster;
import org.apache.streampark.console.core.entity.FlinkEnv;
import org.apache.streampark.console.core.entity.SavePoint;
+import org.apache.streampark.console.core.enums.CheckPointStatus;
import org.apache.streampark.console.core.enums.CheckPointType;
+import org.apache.streampark.console.core.enums.OptionState;
import org.apache.streampark.console.core.mapper.SavePointMapper;
+import org.apache.streampark.console.core.metrics.flink.CheckPoints;
+import org.apache.streampark.console.core.service.ApplicationConfigService;
+import org.apache.streampark.console.core.service.ApplicationLogService;
+import org.apache.streampark.console.core.service.ApplicationService;
+import org.apache.streampark.console.core.service.FlinkClusterService;
import org.apache.streampark.console.core.service.FlinkEnvService;
import org.apache.streampark.console.core.service.SavePointService;
+import org.apache.streampark.console.core.task.CheckpointProcessor;
+import org.apache.streampark.console.core.task.FlinkRESTAPIWatcher;
+import org.apache.streampark.flink.client.FlinkClient;
+import org.apache.streampark.flink.client.bean.SavepointResponse;
+import org.apache.streampark.flink.client.bean.TriggerSavepointRequest;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.metadata.IPage;
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.flink.configuration.CheckpointingOptions;
+import org.apache.flink.configuration.RestOptions;
+import org.apache.flink.runtime.rest.util.RestMapperUtils;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
+import org.jetbrains.annotations.NotNull;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;
+import javax.annotation.Nullable;
+
+import java.net.URI;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
@Slf4j
@Service
@Transactional(propagation = Propagation.SUPPORTS, readOnly = true, rollbackFor = Exception.class)
public class SavePointServiceImpl extends ServiceImpl<SavePointMapper, SavePoint>
implements SavePointService {
+ private static final ObjectMapper OBJECT_MAPPER = RestMapperUtils.getStrictObjectMapper();
+
@Autowired private FlinkEnvService flinkEnvService;
+ @Autowired private ApplicationService appService;
+
+ @Autowired private SavePointService savePointService;
+
+ @Autowired private ApplicationConfigService configService;
+ @Autowired private FlinkClusterService flinkClusterService;
Review Comment:
It is better to have a blank line between current line and the previous line, it is necessary to maintain a good code format
##########
streampark-flink/streampark-flink-client/streampark-flink-client-api/src/main/scala/org/apache/streampark/flink/client/bean/TriggerSavepointRequest.scala:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.streampark.flink.client.bean
+
+import java.util.{Map => JavaMap}
+
+import org.apache.streampark.common.conf.{FlinkVersion, K8sFlinkConfig}
+import org.apache.streampark.common.enums.ExecutionMode
+
+import javax.annotation.Nullable
+
+/** Trigger savepoint request. */
+case class TriggerSavepointRequest(flinkVersion: FlinkVersion,
+ executionMode: ExecutionMode,
+ clusterId: String,
+ jobId: String,
+ customSavePointPath: String,
Review Comment:
i think `savePath` or `savePointPath `is better, we don't need `custom` prefix
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/metrics/flink/CheckPoints.java:
##########
@@ -82,5 +96,20 @@ public String getPath() {
@Data
public static class Latest implements Serializable {
private CheckPoint completed;
+ private CheckPoint savepoint;
+
+ @JsonIgnore
+ public List<CheckPoint> getLatestChkSavepointByIdOrder() {
Review Comment:
how about `getLatestCheckPoint` ? The method name `getLatestChkSavepointByIdOrder` is too long. As a method, it should not expose more internal implementation details. getLatest has expressed the purpose of the method, So I think it is don't necessary to add "ByIdOrder"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1111209488
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +66,32 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
- Application application = applicationService.getById(appId);
+ public void process(Application application, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(application, checkPoint));
+ }
+
+ private void process(Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ String jobID = application.getJobId();
+ Long appId = application.getId();
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, jobID);
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(appId, jobID, checkPoint)) {
+ savepointedCache.put(
+ getCacheIdForSavepoint(appId, jobID, checkPoint.getId()), DEFAULT_FLAG_BYTE);
+ saveSavepoint(
+ checkPoint,
+ application.getId(),
+ latestChkId == null || checkPoint.getId() > latestChkId);
+ return;
+ }
+ if (shouldStoreAsCheckpoint(checkPoint, latestChkId)) {
Review Comment:
```suggestion
if (shouldStoreAsSavepoint(appId, jobID, checkPoint)) {
savepointedCache.put(
getCacheIdForSavepoint(appId, jobID, checkPoint.getId()), DEFAULT_FLAG_BYTE);
saveSavepoint(
checkPoint,
application.getId(),
latestChkId == null || checkPoint.getId() > latestChkId);
return;
}
String cacheId = getCacheIdForCheckpoint(appId, jobID);
Long latestChkId = getLatestCheckpointedId(appId, cacheId);
if (shouldStoreAsCheckpoint(checkPoint, latestChkId)) {
```
These two lines should be moved after the savepoint due to they aren't used for savepoint.
```
String cacheId = getCacheIdForCheckpoint(appId, jobID);
Long latestChkId = getLatestCheckpointedId(appId, cacheId);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys merged pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys merged PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1096471925
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/ApplicationServiceImpl.java:
##########
@@ -1267,8 +1268,14 @@ public void cancel(Application appParam) throws Exception {
savePoint.setAppId(application.getId());
savePoint.setLatest(true);
savePoint.setType(CheckPointType.SAVEPOINT.get());
- savePoint.setTriggerTime(now);
+ savePoint.setTriggerTime(triggerTime);
savePoint.setCreateTime(now);
+ savePoint.setJobId(cancelRequest.jobId());
+ savePoint.setEndTime(now);
+ savePoint.setClusterId(application.getFlinkClusterId());
+ if (appParam.getExecutionMode() != null) {
+ savePoint.setExecutionMode(ExecutionMode.of(appParam.getExecutionMode()).name());
Review Comment:
`appParam.getExecutionMode()` is also string, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1096473616
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -108,6 +112,23 @@ public void process(Long appId, CheckPoints checkPoints) {
}
}
+ private boolean shouldProcessFailedTrigger(
+ CheckPoints.CheckPoint checkPoint, Application application, CheckPointStatus status) {
+ return CheckPointStatus.FAILED.equals(status)
+ && !checkPoint.getIsSavepoint()
+ && application.cpFailedTrigger();
+ }
+
+ private boolean checkpointNeedStore(
+ Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ LambdaQueryWrapper<SavePoint> queryWrapper =
Review Comment:
I mean we still use the old logic.
The code of the master branch determines whether checkpoint/savepoint needs to be stored in the database according to the cache, and I think it is enough for now. Your code will access to the database frequently, it isn't friendly to database.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#issuecomment-1417950591
@1996fanrui Thank so much for the review.
Updated it~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1097552846
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -114,6 +127,41 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application app = appService.getById(appId);
+ AssertUtils.notNull(app, String.format("The application %s doesn't exist.", appId));
+ String trackingUrl = app.getJobManagerUrl();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(trackingUrl),
+ String.format("The flink trackingUrl for app[%s] isn't available.", appId));
+ String jobId = app.getJobId();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(jobId),
+ String.format("The jobId of application[%s] is absent.", appId));
+ String triggerId = triggerSavepoint(trackingUrl, jobId, savepointPath);
+ log.info("Request savepoint successful in triggerId {}", triggerId);
+ }
+
+ private String triggerSavepoint(
+ String trackingUrl, String jobId, @Nullable String savepointPath) {
+ String response = null;
+ try {
+ String url = String.format("%s/jobs/%s/savepoints", trackingUrl, jobId);
+ String postBody =
+ OBJECT_MAPPER.writeValueAsString(new SavepointTriggerRequestBody(savepointPath, false));
+ response = HttpClientUtils.httpPostRequest(url, postBody);
+ TriggerResponse triggerResponse = OBJECT_MAPPER.readValue(response, TriggerResponse.class);
+ return triggerResponse.getTriggerId().toString();
+ } catch (Exception e) {
+ throw new RuntimeException(
Review Comment:
`AlertException`: notify users of some info by WeChat,DingTalk, Msg...,
`ApiAlertException`: some message will be notify to user through the front-end pop-up window of streampark,
so, we should use the `ApiAlertException` here, not `AlertException `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1103636831
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -41,9 +44,16 @@
@Component
public class CheckpointProcessor {
+ private static final Byte DEFAULT_FLAG_BYTE = Byte.valueOf("0");
+ private static final Integer SAVEPOINT_CACHE_HOUR = 1;
+
private final Cache<String, Long> checkPointCache =
Caffeine.newBuilder().expireAfterAccess(1, TimeUnit.DAYS).build();
+ /** Cache to store the savepoint path if be stored in the db. */
+ private final Cache<String, Byte> savepointedCache =
Review Comment:
> Are you worried about `execution.checkpointing.max-concurrent-checkpoints > 1`?
Yes, the introduced lines is used to adapt for the setting case only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1104578765
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +62,31 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
+ public void process(Long appId, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(appId, checkPoint));
+ }
+
+ private void process(Long appId, @Nonnull CheckPoints.CheckPoint checkPoint) {
Application application = applicationService.getById(appId);
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, application.getJobId());
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(checkPoint, latestChkId)) {
Review Comment:
If in this cased you described, the logic will go to `! isStoredSavepoint`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1104521381
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +62,31 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
+ public void process(Long appId, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(appId, checkPoint));
+ }
+
+ private void process(Long appId, @Nonnull CheckPoints.CheckPoint checkPoint) {
Application application = applicationService.getById(appId);
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, application.getJobId());
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(checkPoint, latestChkId)) {
+ savepointedCache.put(checkPoint.getExternalPath(), DEFAULT_FLAG_BYTE);
+ if ((latestChkId == null || checkPoint.getId() > latestChkId)) {
+ checkPointCache.put(cacheId, checkPoint.getId());
Review Comment:
If `savepointedCache` is necessary, why do you store the savepoint to checkPointCache? Could it just be stored in `savepointedCache`?
In fact, the logic is a bit more complicated now, it's better to simplify it. It reduces the chance of bugs and is easier for other developers to understand.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +62,31 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
+ public void process(Long appId, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(appId, checkPoint));
+ }
+
+ private void process(Long appId, @Nonnull CheckPoints.CheckPoint checkPoint) {
Application application = applicationService.getById(appId);
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, application.getJobId());
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(checkPoint, latestChkId)) {
+ savepointedCache.put(checkPoint.getExternalPath(), DEFAULT_FLAG_BYTE);
Review Comment:
Why using the `DEFAULT_FLAG_BYTE` as the value?
Why don't using the similar logic to checkpointCache that using the cacheId as the key and checkpointId as the value?
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +62,31 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
+ public void process(Long appId, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(appId, checkPoint));
+ }
+
+ private void process(Long appId, @Nonnull CheckPoints.CheckPoint checkPoint) {
Application application = applicationService.getById(appId);
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, application.getJobId());
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(checkPoint, latestChkId)) {
Review Comment:
> However, if the savepoint store-logic follow checkpoint, here would be some bug-case. If the triggered savepoint is completed after a completed checkpoint whose chk-id is bigger than the savepoint, the savepoint will be discard based on the current logic.
When it happened, the latestChkId is bigger than the savepoint. So `shouldStoreAsSavepoint(checkPoint, latestChkId)` returns false, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1105267724
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +62,31 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
+ public void process(Long appId, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(appId, checkPoint));
+ }
+
+ private void process(Long appId, @Nonnull CheckPoints.CheckPoint checkPoint) {
Application application = applicationService.getById(appId);
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, application.getJobId());
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(checkPoint, latestChkId)) {
+ savepointedCache.put(checkPoint.getExternalPath(), DEFAULT_FLAG_BYTE);
Review Comment:
> Why can't? Doesn't it have TriggerTime? If triggerTime < current time - TTL, it should be discarded, right?
>
> >
In this assumption, the same one savepoint from REST-API response perhaps store multiple times in the TTL range..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1109937496
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/metrics/flink/CheckPoints.java:
##########
@@ -82,5 +96,20 @@ public String getPath() {
@Data
public static class Latest implements Serializable {
private CheckPoint completed;
+ private CheckPoint savepoint;
+
+ @JsonIgnore
+ public List<CheckPoint> getLatestCheckpoint() {
+ List<CheckPoint> checkPoints = new ArrayList<>();
+ if (Objects.nonNull(completed)) {
+ checkPoints.add(completed);
+ }
+ if (Objects.nonNull(savepoint)) {
+ checkPoints.add(savepoint);
+ }
+ return checkPoints.stream()
+ .sorted(Comparator.comparingLong(chk -> chk.id))
Review Comment:
Yes, I'll update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#issuecomment-1433119050
> hi RocMarshal please resolve conflicts.
@wolfboys Thank you for the review.
I'll resolve it .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#issuecomment-1407567384
> Hi @RocMarshal , thanks for your contribution.
>
> > @1996fanrui If you are free, can you review it first?
>
> I'm on vacation these days, I can start review late next week.
@1996fanrui Enjoy it~.
@wolfboys Thank you for your support on the review .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1111214415
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +66,32 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
- Application application = applicationService.getById(appId);
+ public void process(Application application, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(application, checkPoint));
+ }
+
+ private void process(Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ String jobID = application.getJobId();
+ Long appId = application.getId();
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, jobID);
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(appId, jobID, checkPoint)) {
+ savepointedCache.put(
+ getCacheIdForSavepoint(appId, jobID, checkPoint.getId()), DEFAULT_FLAG_BYTE);
+ saveSavepoint(
+ checkPoint,
+ application.getId(),
+ latestChkId == null || checkPoint.getId() > latestChkId);
+ return;
+ }
+ if (shouldStoreAsCheckpoint(checkPoint, latestChkId)) {
Review Comment:
thx for the comments. let me check it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1110522915
##########
streampark-flink/streampark-flink-client/streampark-flink-client-core/src/main/scala/org/apache/streampark/flink/client/impl/KubernetesNativeSessionSubmit.scala:
##########
@@ -212,4 +212,9 @@ object KubernetesNativeSessionSubmit extends KubernetesNativeSubmitTrait with Lo
Utils.close(kubeClient)
}
}
+
+ override def doTriggerSavepoint(request: TriggerSavepointRequest, flinkConfig: Configuration): SavepointResponse = {
+ flinkConfig.safeSet(DeploymentOptions.TARGET, ExecutionMode.KUBERNETES_NATIVE_APPLICATION.getName)
Review Comment:
> Is it a bug?
Confirmed with @wolfboys offline. It's a bug. I fixed it.
PTAL. :)
Thank you @wolfboys @1996fanrui
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1097230372
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/ApplicationServiceImpl.java:
##########
@@ -1258,7 +1259,7 @@ public void cancel(Application appParam) throws Exception {
savePoint.setAppId(application.getId());
savePoint.setLatest(true);
savePoint.setType(CheckPointType.SAVEPOINT.get());
- savePoint.setTriggerTime(now);
Review Comment:

##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -114,6 +127,41 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application app = appService.getById(appId);
+ AssertUtils.notNull(app, String.format("The application %s doesn't exist.", appId));
+ String trackingUrl = app.getJobManagerUrl();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(trackingUrl),
+ String.format("The flink trackingUrl for app[%s] isn't available.", appId));
+ String jobId = app.getJobId();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(jobId),
+ String.format("The jobId of application[%s] is absent.", appId));
+ String triggerId = triggerSavepoint(trackingUrl, jobId, savepointPath);
+ log.info("Request savepoint successful in triggerId {}", triggerId);
+ }
+
+ private String triggerSavepoint(
+ String trackingUrl, String jobId, @Nullable String savepointPath) {
+ String response = null;
+ try {
+ String url = String.format("%s/jobs/%s/savepoints", trackingUrl, jobId);
+ String postBody =
+ OBJECT_MAPPER.writeValueAsString(new SavepointTriggerRequestBody(savepointPath, false));
+ response = HttpClientUtils.httpPostRequest(url, postBody);
+ TriggerResponse triggerResponse = OBJECT_MAPPER.readValue(response, TriggerResponse.class);
+ return triggerResponse.getTriggerId().toString();
+ } catch (Exception e) {
+ throw new RuntimeException(
Review Comment:
I see @RocMarshal changed it to `ApiException`, @wolfboys , could you please help check again? Should we use the `ApiAlertException` or `ApiException`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1103616775
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/ApplicationServiceImpl.java:
##########
@@ -1258,7 +1259,7 @@ public void cancel(Application appParam) throws Exception {
savePoint.setAppId(application.getId());
savePoint.setLatest(true);
savePoint.setType(CheckPointType.SAVEPOINT.get());
- savePoint.setTriggerTime(now);
Review Comment:
No, is the `now` used after your PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1103628964
##########
streampark-flink/streampark-flink-client/streampark-flink-client-core/src/main/scala/org/apache/streampark/flink/client/FlinkClientHandler.scala:
##########
@@ -36,6 +36,19 @@ object FlinkClientHandler {
}
}
+ def triggerSavepoint(savepointRequest: TriggerSavepointRequest): SavepointResponse = {
+ savepointRequest.executionMode match {
+ case ExecutionMode.LOCAL => LocalSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.REMOTE => RemoteSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.YARN_APPLICATION => YarnApplicationSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.YARN_SESSION => YarnSessionSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.YARN_PER_JOB | ExecutionMode.YARN_SESSION => YarnPerJobSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.KUBERNETES_NATIVE_SESSION => KubernetesNativeSessionSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.KUBERNETES_NATIVE_APPLICATION => KubernetesNativeApplicationSubmit.triggerSavepoint(savepointRequest)
+ case _ => throw new UnsupportedOperationException(s"Unsupported ${savepointRequest.executionMode} Submit ")
+ }
+ }
+
Review Comment:
Indeed, as @1996fanrui you said. The current design logic makes the code inevitably bloated.
We can start with a new Issue to sort out the desired effect of the refactoring, the problems at this stage, and the best way to refactor.
CC @wolfboys
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#issuecomment-1416647298
> @1996fanrui Thx so much for the comments. I'll update it after the comments discussed.
Hi @RocMarshal , thanks for your quick feedback.
From your feedback, I found two main problems:
- You added a lot of code that is not currently needed in this PR, such as: status, end_time, execution_mode, cluster_id, etc. And we can add them if they're needed in the future.
- Modify the logic of `CheckpointProcessor` as little as possible. The code of the master branch determines whether checkpoint/savepoint needs to be stored in the database according to the cache, and I think it is enough for now. Your code will access to the database frequently, it isn't friendly to database.
Overall, this PR should only contain logic to trigger savepoint and fetch savepoint list.
Also, I want to check with you: the `completed` of the `Latest` just includes checkpoint, and it doesn't include savepoint, right?
<img width="959" alt="image" src="https://user-images.githubusercontent.com/38427477/216745556-97ce34d3-38ac-42b8-81ea-2ce22ec4c966.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1105141199
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +62,31 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
+ public void process(Long appId, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(appId, checkPoint));
+ }
+
+ private void process(Long appId, @Nonnull CheckPoints.CheckPoint checkPoint) {
Application application = applicationService.getById(appId);
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, application.getJobId());
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(checkPoint, latestChkId)) {
+ savepointedCache.put(checkPoint.getExternalPath(), DEFAULT_FLAG_BYTE);
+ if ((latestChkId == null || checkPoint.getId() > latestChkId)) {
+ checkPointCache.put(cacheId, checkPoint.getId());
Review Comment:
The checkpoint works well if we don't store savepoint into the check point cache.
When start job from cp or sp, we can sort them by trigger time to choose the latest one. Is there any bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1104623853
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +62,31 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
+ public void process(Long appId, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(appId, checkPoint));
+ }
+
+ private void process(Long appId, @Nonnull CheckPoints.CheckPoint checkPoint) {
Application application = applicationService.getById(appId);
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, application.getJobId());
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(checkPoint, latestChkId)) {
+ savepointedCache.put(checkPoint.getExternalPath(), DEFAULT_FLAG_BYTE);
Review Comment:
> Why don't using the similar logic to checkpointCache that using the cacheId as the key and checkpointId as the value?
We can't judge the savepoint if be stored out of TTL. From the point of view, it doesn't matter what type of value is stored. But the appId + jobId if shorter than savepoint path as cache key mostly.
I‘m hesitating whether we should take savepoint as more important than checkpoint. If we treat each other equally, on the contrary, this part of logic will become simple to share the cache with checkpoint
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1094488180
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -210,12 +210,19 @@ drop table if exists `t_flink_savepoint`;
create table `t_flink_savepoint` (
`id` bigint not null auto_increment,
`app_id` bigint not null,
+ `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
Review Comment:
Why need the `job_id`?
##########
streampark-console/streampark-console-service/src/main/assembly/script/data/mysql-data.sql:
##########
@@ -117,6 +117,7 @@ insert into `t_menu` values (100066, 100015, 'view', null, null, 'app:view', nul
insert into `t_menu` values (100067, 100053, 'view', NULL, NULL, 'variable:view', NULL, '1', 1, null, now(), now());
insert into `t_menu` values (100068, 100033, 'view', null, null, 'setting:view', null, '1', 1, null, now(), now());
insert into `t_menu` values (100069, 100053, 'depend view', null, null, 'variable:depend_apps', null, '1', 1, NULL, now(), now());
+insert into `t_menu` values (100070, 100015, 'savepoint trigger', null, null, 'savepoint:triggerSavepoint', null, '1', 1, null, now(), now());
Review Comment:
How about change the `savepoint:triggerSavepoint` to `savepoint:trigger`? It is clear to trigger savepoint.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -108,6 +112,23 @@ public void process(Long appId, CheckPoints checkPoints) {
}
}
+ private boolean shouldProcessFailedTrigger(
+ CheckPoints.CheckPoint checkPoint, Application application, CheckPointStatus status) {
+ return CheckPointStatus.FAILED.equals(status)
+ && !checkPoint.getIsSavepoint()
+ && application.cpFailedTrigger();
+ }
+
+ private boolean checkpointNeedStore(
+ Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ LambdaQueryWrapper<SavePoint> queryWrapper =
+ new LambdaQueryWrapper<SavePoint>()
+ .eq(SavePoint::getAppId, application.getAppId())
+ .eq(SavePoint::getJobId, application.getJobId())
Review Comment:
We don't suggest add the jobId here, it's better to using the `instance_id` in the future, `instance_id` is more intuitively.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -108,6 +112,23 @@ public void process(Long appId, CheckPoints checkPoints) {
}
}
+ private boolean shouldProcessFailedTrigger(
+ CheckPoints.CheckPoint checkPoint, Application application, CheckPointStatus status) {
+ return CheckPointStatus.FAILED.equals(status)
+ && !checkPoint.getIsSavepoint()
+ && application.cpFailedTrigger();
+ }
+
+ private boolean checkpointNeedStore(
+ Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ LambdaQueryWrapper<SavePoint> queryWrapper =
Review Comment:
We should check the `checkPointCache` first, right?
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -210,12 +210,19 @@ drop table if exists `t_flink_savepoint`;
create table `t_flink_savepoint` (
`id` bigint not null auto_increment,
`app_id` bigint not null,
+ `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `status` VARCHAR(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'COMPLETED',
Review Comment:
I see the checkpoint or savepoint is saved only when `status` is 'COMPLETED', right?
If yes, why need the `status`?
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -210,12 +210,19 @@ drop table if exists `t_flink_savepoint`;
create table `t_flink_savepoint` (
`id` bigint not null auto_increment,
`app_id` bigint not null,
+ `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `status` VARCHAR(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'COMPLETED',
`chk_id` bigint default null,
`type` tinyint default null,
- `path` varchar(255) collate utf8mb4_general_ci default null,
+ `path` varchar(2048) collate utf8mb4_general_ci default null,
`latest` tinyint not null default 1,
`trigger_time` datetime default null,
+ `execution_mode` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
Review Comment:
What does the `execution_mode` mean here? And why need it here?
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/ApplicationServiceImpl.java:
##########
@@ -1267,8 +1268,14 @@ public void cancel(Application appParam) throws Exception {
savePoint.setAppId(application.getId());
savePoint.setLatest(true);
savePoint.setType(CheckPointType.SAVEPOINT.get());
- savePoint.setTriggerTime(now);
+ savePoint.setTriggerTime(triggerTime);
savePoint.setCreateTime(now);
+ savePoint.setJobId(cancelRequest.jobId());
+ savePoint.setEndTime(now);
+ savePoint.setClusterId(application.getFlinkClusterId());
+ if (appParam.getExecutionMode() != null) {
+ savePoint.setExecutionMode(ExecutionMode.of(appParam.getExecutionMode()).name());
Review Comment:
`ExecutionMode.of(appParam.getExecutionMode()).name()` == `appParam.getExecutionMode()`, right?
If yes, how about `savePoint.setExecutionMode(appParam.getExecutionMode())`?
And the `if (appParam.getExecutionMode() != null) ` can be removed.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/FlinkRESTAPIWatcher.java:
##########
@@ -357,7 +357,9 @@ private void handleCheckPoints(Application application) throws Exception {
FlinkCluster flinkCluster = getFlinkCluster(application);
CheckPoints checkPoints = httpCheckpoints(application, flinkCluster);
if (checkPoints != null) {
- checkpointProcessor.process(application.getId(), checkPoints);
+ checkPoints
+ .getLatestChkAndSavepointWithTimeAscOrder()
Review Comment:
Why `getLatestChkAndSavepointWithTimeAscOrder ` return a list? And why need process all checkpoints and savepoints? The latest checkpoint and latest savepoint should be enough, right?
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -36,19 +38,30 @@
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.flink.runtime.rest.handler.async.TriggerResponse;
+import org.apache.flink.runtime.rest.messages.job.savepoints.SavepointTriggerRequestBody;
+import org.apache.flink.runtime.rest.util.RestMapperUtils;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;
+import javax.annotation.Nullable;
+
@Slf4j
@Service
@Transactional(propagation = Propagation.SUPPORTS, readOnly = true, rollbackFor = Exception.class)
public class SavePointServiceImpl extends ServiceImpl<SavePointMapper, SavePoint>
implements SavePointService {
+ private static final ObjectMapper objMapper = RestMapperUtils.getStrictObjectMapper();
Review Comment:
The static field name should be uppercase: `OBJECT_MAPPER`
##########
streampark-console/streampark-console-service/src/main/assembly/script/upgrade/ISSUE-2192/schema/2192-mysql.sql:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+alter table `t_flink_savepoint` add column `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL after `app_id`;
Review Comment:
The upgrade sql should be moved to [here](https://github.com/apache/incubator-streampark/tree/dev/streampark-console/streampark-console-service/src/main/assembly/script/upgrade/mysql). And file name should be `2.1.0.sql`.
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -210,12 +210,19 @@ drop table if exists `t_flink_savepoint`;
create table `t_flink_savepoint` (
`id` bigint not null auto_increment,
`app_id` bigint not null,
+ `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `status` VARCHAR(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'COMPLETED',
`chk_id` bigint default null,
`type` tinyint default null,
- `path` varchar(255) collate utf8mb4_general_ci default null,
+ `path` varchar(2048) collate utf8mb4_general_ci default null,
`latest` tinyint not null default 1,
`trigger_time` datetime default null,
+ `execution_mode` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `cluster_id` bigint DEFAULT NULL,
+ `end_time` datetime DEFAULT NULL,
Review Comment:
Why need the `end_time `?
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -114,6 +127,42 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public Boolean triggerSavepoint(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application app = appService.getById(appId);
+ AssertUtils.notNull(app, String.format("The application %s doesn't exist.", appId));
+ String trackingUrl = app.getJobManagerUrl();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(trackingUrl),
+ String.format("The flink trackingUrl for app[%s] isn't available.", appId));
+ String jobId = app.getJobId();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(jobId),
+ String.format("The jobId of application[%s] is absent.", appId));
+ String triggerId = triggerSavepoint(trackingUrl, jobId, savepointPath);
+ log.info("Request savepoint successful in triggerId {}", triggerId);
+ return true;
Review Comment:
This method either returns true or throws an exception. Never returns false, so the return value is meaningless and can be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1094519802
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -210,12 +210,19 @@ drop table if exists `t_flink_savepoint`;
create table `t_flink_savepoint` (
`id` bigint not null auto_increment,
`app_id` bigint not null,
+ `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `status` VARCHAR(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'COMPLETED',
`chk_id` bigint default null,
`type` tinyint default null,
- `path` varchar(255) collate utf8mb4_general_ci default null,
+ `path` varchar(2048) collate utf8mb4_general_ci default null,
`latest` tinyint not null default 1,
`trigger_time` datetime default null,
+ `execution_mode` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
Review Comment:
Why need the `execution_mode` here?
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -210,12 +210,19 @@ drop table if exists `t_flink_savepoint`;
create table `t_flink_savepoint` (
`id` bigint not null auto_increment,
`app_id` bigint not null,
+ `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `status` VARCHAR(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'COMPLETED',
`chk_id` bigint default null,
`type` tinyint default null,
- `path` varchar(255) collate utf8mb4_general_ci default null,
+ `path` varchar(2048) collate utf8mb4_general_ci default null,
`latest` tinyint not null default 1,
`trigger_time` datetime default null,
+ `execution_mode` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `cluster_id` bigint DEFAULT NULL,
Review Comment:
And why need the cluster_id?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1094608384
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -210,12 +210,19 @@ drop table if exists `t_flink_savepoint`;
create table `t_flink_savepoint` (
`id` bigint not null auto_increment,
`app_id` bigint not null,
+ `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
Review Comment:
It could be represented that the savepoint status is close to the jobId instead the big grain scope `application`.
However, It's not precise enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1095825611
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -108,6 +112,23 @@ public void process(Long appId, CheckPoints checkPoints) {
}
}
+ private boolean shouldProcessFailedTrigger(
+ CheckPoints.CheckPoint checkPoint, Application application, CheckPointStatus status) {
+ return CheckPointStatus.FAILED.equals(status)
+ && !checkPoint.getIsSavepoint()
+ && application.cpFailedTrigger();
+ }
+
+ private boolean checkpointNeedStore(
+ Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ LambdaQueryWrapper<SavePoint> queryWrapper =
Review Comment:
did you mean we should judge if the chk/savepoint id exist in the cache?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1094617097
##########
streampark-console/streampark-console-service/src/main/assembly/script/upgrade/ISSUE-2192/schema/2192-mysql.sql:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+alter table `t_flink_savepoint` add column `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL after `app_id`;
Review Comment:
But now, the developers couldn't know the next release version in time.
There're two optional idea:
- The community supply the release versions arrange ahead of schedule, developers will follow the rule to coding.
- The delta ddl & ddl could be coded with the issue-id. The PIC of the next release sort it based on the version before the release.
Any suggestion is appreciated.
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -210,12 +210,19 @@ drop table if exists `t_flink_savepoint`;
create table `t_flink_savepoint` (
`id` bigint not null auto_increment,
`app_id` bigint not null,
+ `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `status` VARCHAR(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'COMPLETED',
Review Comment:
It isn't appropriate to introduce the field in the Issue.
The platform doesn't collect the failed savepoint/chk into the db.
Maybe it's a good feature to record the exceptionally savepoint/chk for job monitor.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -114,6 +127,42 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public Boolean triggerSavepoint(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application app = appService.getById(appId);
+ AssertUtils.notNull(app, String.format("The application %s doesn't exist.", appId));
+ String trackingUrl = app.getJobManagerUrl();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(trackingUrl),
+ String.format("The flink trackingUrl for app[%s] isn't available.", appId));
+ String jobId = app.getJobId();
+ AssertUtils.state(
+ StringUtils.isNotEmpty(jobId),
+ String.format("The jobId of application[%s] is absent.", appId));
+ String triggerId = triggerSavepoint(trackingUrl, jobId, savepointPath);
+ log.info("Request savepoint successful in triggerId {}", triggerId);
+ return true;
Review Comment:
nice ~
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -210,12 +210,19 @@ drop table if exists `t_flink_savepoint`;
create table `t_flink_savepoint` (
`id` bigint not null auto_increment,
`app_id` bigint not null,
+ `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `status` VARCHAR(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'COMPLETED',
`chk_id` bigint default null,
`type` tinyint default null,
- `path` varchar(255) collate utf8mb4_general_ci default null,
+ `path` varchar(2048) collate utf8mb4_general_ci default null,
`latest` tinyint not null default 1,
`trigger_time` datetime default null,
+ `execution_mode` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
Review Comment:
It's bunded with `status`, which could help user to trace the exceptional savepoint/chk `execution` environment.
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -210,12 +210,19 @@ drop table if exists `t_flink_savepoint`;
create table `t_flink_savepoint` (
`id` bigint not null auto_increment,
`app_id` bigint not null,
+ `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `status` VARCHAR(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'COMPLETED',
`chk_id` bigint default null,
`type` tinyint default null,
- `path` varchar(255) collate utf8mb4_general_ci default null,
+ `path` varchar(2048) collate utf8mb4_general_ci default null,
`latest` tinyint not null default 1,
`trigger_time` datetime default null,
+ `execution_mode` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `cluster_id` bigint DEFAULT NULL,
Review Comment:
be same as above.
##########
streampark-console/streampark-console-service/src/main/assembly/script/schema/mysql-schema.sql:
##########
@@ -210,12 +210,19 @@ drop table if exists `t_flink_savepoint`;
create table `t_flink_savepoint` (
`id` bigint not null auto_increment,
`app_id` bigint not null,
+ `job_id` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `status` VARCHAR(255) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'COMPLETED',
`chk_id` bigint default null,
`type` tinyint default null,
- `path` varchar(255) collate utf8mb4_general_ci default null,
+ `path` varchar(2048) collate utf8mb4_general_ci default null,
`latest` tinyint not null default 1,
`trigger_time` datetime default null,
+ `execution_mode` VARCHAR(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
+ `cluster_id` bigint DEFAULT NULL,
+ `end_time` datetime DEFAULT NULL,
Review Comment:
It could be used to calculate the chk/savepoint time cost.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -108,6 +112,23 @@ public void process(Long appId, CheckPoints checkPoints) {
}
}
+ private boolean shouldProcessFailedTrigger(
+ CheckPoints.CheckPoint checkPoint, Application application, CheckPointStatus status) {
+ return CheckPointStatus.FAILED.equals(status)
+ && !checkPoint.getIsSavepoint()
+ && application.cpFailedTrigger();
+ }
+
+ private boolean checkpointNeedStore(
+ Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ LambdaQueryWrapper<SavePoint> queryWrapper =
Review Comment:
A minor confusion for me form the comments.
Does the minor refactor cause the logic loss or the original implementation?
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -108,6 +112,23 @@ public void process(Long appId, CheckPoints checkPoints) {
}
}
+ private boolean shouldProcessFailedTrigger(
+ CheckPoints.CheckPoint checkPoint, Application application, CheckPointStatus status) {
+ return CheckPointStatus.FAILED.equals(status)
+ && !checkPoint.getIsSavepoint()
+ && application.cpFailedTrigger();
+ }
+
+ private boolean checkpointNeedStore(
+ Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ LambdaQueryWrapper<SavePoint> queryWrapper =
+ new LambdaQueryWrapper<SavePoint>()
+ .eq(SavePoint::getAppId, application.getAppId())
+ .eq(SavePoint::getJobId, application.getJobId())
Review Comment:
yes, I get it.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/FlinkRESTAPIWatcher.java:
##########
@@ -357,7 +357,9 @@ private void handleCheckPoints(Application application) throws Exception {
FlinkCluster flinkCluster = getFlinkCluster(application);
CheckPoints checkPoints = httpCheckpoints(application, flinkCluster);
if (checkPoints != null) {
- checkpointProcessor.process(application.getId(), checkPoints);
+ checkPoints
+ .getLatestChkAndSavepointWithTimeAscOrder()
Review Comment:
The original intention is that get chk & savepoint from`latest` with order, then records the info into db. The latest savepoint or chk will be tagged with 'latest'.
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/ApplicationServiceImpl.java:
##########
@@ -1267,8 +1268,14 @@ public void cancel(Application appParam) throws Exception {
savePoint.setAppId(application.getId());
savePoint.setLatest(true);
savePoint.setType(CheckPointType.SAVEPOINT.get());
- savePoint.setTriggerTime(now);
+ savePoint.setTriggerTime(triggerTime);
savePoint.setCreateTime(now);
+ savePoint.setJobId(cancelRequest.jobId());
+ savePoint.setEndTime(now);
+ savePoint.setClusterId(application.getFlinkClusterId());
+ if (appParam.getExecutionMode() != null) {
+ savePoint.setExecutionMode(ExecutionMode.of(appParam.getExecutionMode()).name());
Review Comment:
good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1094604965
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -36,19 +38,30 @@
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.flink.runtime.rest.handler.async.TriggerResponse;
+import org.apache.flink.runtime.rest.messages.job.savepoints.SavepointTriggerRequestBody;
+import org.apache.flink.runtime.rest.util.RestMapperUtils;
+import org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;
+import javax.annotation.Nullable;
+
@Slf4j
@Service
@Transactional(propagation = Propagation.SUPPORTS, readOnly = true, rollbackFor = Exception.class)
public class SavePointServiceImpl extends ServiceImpl<SavePointMapper, SavePoint>
implements SavePointService {
+ private static final ObjectMapper objMapper = RestMapperUtils.getStrictObjectMapper();
Review Comment:
The variable field could be camal-case style in the private scope based on the current code style.
I'll update it if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1103649241
##########
streampark-flink/streampark-flink-client/streampark-flink-client-core/src/main/scala/org/apache/streampark/flink/client/FlinkClientHandler.scala:
##########
@@ -36,6 +36,19 @@ object FlinkClientHandler {
}
}
+ def triggerSavepoint(savepointRequest: TriggerSavepointRequest): SavepointResponse = {
+ savepointRequest.executionMode match {
+ case ExecutionMode.LOCAL => LocalSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.REMOTE => RemoteSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.YARN_APPLICATION => YarnApplicationSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.YARN_SESSION => YarnSessionSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.YARN_PER_JOB | ExecutionMode.YARN_SESSION => YarnPerJobSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.KUBERNETES_NATIVE_SESSION => KubernetesNativeSessionSubmit.triggerSavepoint(savepointRequest)
+ case ExecutionMode.KUBERNETES_NATIVE_APPLICATION => KubernetesNativeApplicationSubmit.triggerSavepoint(savepointRequest)
+ case _ => throw new UnsupportedOperationException(s"Unsupported ${savepointRequest.executionMode} Submit ")
+ }
+ }
+
Review Comment:
> Indeed, as @1996fanrui you said. The current design logic makes the code inevitably bloated. We can start with a new Issue to sort out the desired effect of the refactoring, the problems at this stage, and the best way to refactor. CC @wolfboys
yes. we can refactor it in next PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1108460337
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -171,6 +218,182 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public String getSavePointPath(Application appParam) throws Exception {
+ Application application = applicationService.getById(appParam.getId());
+
+ // 1) properties have the highest priority, read the properties are set: -Dstate.savepoints.dir
+ String savepointPath =
+ FlinkClient.extractDynamicPropertiesAsJava(application.getDynamicProperties())
+ .get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+
+ // Application conf configuration has the second priority. If it is a streampark|flinksql type
+ // task,
+ // see if Application conf is configured when the task is defined, if checkpoints are configured
+ // and enabled,
+ // read `state.savepoints.dir`
+ if (StringUtils.isBlank(savepointPath)) {
+ if (application.isStreamParkJob() || application.isFlinkSqlJob()) {
+ ApplicationConfig applicationConfig = configService.getEffective(application.getId());
+ if (applicationConfig != null) {
+ Map<String, String> map = applicationConfig.readConfig();
+ if (FlinkUtils.isCheckpointEnabled(map)) {
+ savepointPath = map.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+ }
+ }
+
+ // 3) If the savepoint is not obtained above, try to obtain the savepoint path according to the
+ // deployment type (remote|on yarn)
+ if (StringUtils.isBlank(savepointPath)) {
+ // 3.1) At the remote mode, request the flink webui interface to get the savepoint path
+ if (ExecutionMode.isRemoteMode(application.getExecutionMode())) {
+ FlinkCluster cluster = flinkClusterService.getById(application.getFlinkClusterId());
+ Utils.required(
+ cluster != null,
+ String.format(
+ "The clusterId=%s cannot be find, maybe the clusterId is wrong or "
+ + "the cluster has been deleted. Please contact the Admin.",
+ application.getFlinkClusterId()));
+ Map<String, String> config = cluster.getFlinkConfig();
+ if (!config.isEmpty()) {
+ savepointPath = config.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ } else {
+ // 3.2) At the yarn or k8s mode, then read the savepoint in flink-conf.yml in the bound
+ // flink
+ FlinkEnv flinkEnv = flinkEnvService.getById(application.getVersionId());
+ savepointPath =
+ flinkEnv.convertFlinkYamlAsMap().get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+
+ return savepointPath;
+ }
+
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application application = applicationService.getById(appId);
+
+ FlinkRESTAPIWatcher.addSavepoint(application.getId());
+
+ application.setOptionState(OptionState.SAVEPOINTING.getValue());
+ application.setOptionTime(new Date());
+ this.applicationService.updateById(application);
+
+ FlinkEnv flinkEnv = flinkEnvService.getById(application.getVersionId());
+
+ // infer savepoint
+ String customSavepoint = this.getFinalSavepointDir(savepointPath, application);
+
+ FlinkCluster cluster = flinkClusterService.getById(application.getFlinkClusterId());
+ String clusterId = getClusterId(application, cluster);
+
+ Map<String, Object> properties = this.tryGetRestProps(application, cluster);
+
+ TriggerSavepointRequest request =
+ new TriggerSavepointRequest(
+ flinkEnv.getFlinkVersion(),
+ ExecutionMode.of(application.getExecutionMode()),
+ clusterId,
+ application.getJobId(),
+ customSavepoint,
+ application.getK8sNamespace(),
+ properties);
+
+ CompletableFuture<SavepointResponse> savepointFuture =
+ CompletableFuture.supplyAsync(() -> FlinkClient.triggerSavepoint(request), executorService);
+
+ handleSavepointResponseFuture(application, savepointFuture);
+ }
+
+ private void handleSavepointResponseFuture(
+ Application application, CompletableFuture<SavepointResponse> savepointFuture) {
+ CompletableFutureUtils.runTimeout(
+ savepointFuture,
+ 10L,
+ TimeUnit.MINUTES,
+ savepointResponse -> {
+ if (savepointResponse != null && savepointResponse.savePointDir() != null) {
+ String savePointDir = savepointResponse.savePointDir();
+ log.info("Request savepoint successful, savepointDir: {}", savePointDir);
+ }
+ },
+ e -> {
+ log.error("Trigger savepoint for flink job failed.", e);
+ ApplicationLog log = new ApplicationLog();
+ log.setAppId(application.getId());
+ log.setYarnAppId(application.getClusterId());
+ log.setOptionTime(new Date());
+ String exception = Utils.stringifyException(e);
+ log.setException(exception);
+ if (!(e instanceof TimeoutException)) {
+ log.setSuccess(false);
+ }
+ applicationLogService.save(log);
+ })
+ .whenComplete(
+ (t, e) -> {
+ application.setOptionState(OptionState.NONE.getValue());
+ application.setOptionTime(new Date());
+ applicationService.update(application);
+ });
+ }
+
+ private String getFinalSavepointDir(@Nullable String savepointPath, Application application) {
+ String result = savepointPath;
+ if (StringUtils.isEmpty(savepointPath)) {
+ try {
+ result = this.getSavePointPath(application);
+ } catch (Exception e) {
+ throw new ApiAlertException(
+ "Error in getting savepoint path for triggering savepoint for app "
+ + application.getId(),
+ e);
+ }
+ }
+ return result;
+ }
+
+ @NotNull
+ private Map<String, Object> tryGetRestProps(Application application, FlinkCluster cluster) {
+ Map<String, Object> properties = new HashMap<>();
+
+ if (ExecutionMode.isRemoteMode(application.getExecutionModeEnum())) {
+ Utils.required(
Review Comment:
Utils.notNull(cluster, msg...)
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -171,6 +218,182 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public String getSavePointPath(Application appParam) throws Exception {
+ Application application = applicationService.getById(appParam.getId());
+
+ // 1) properties have the highest priority, read the properties are set: -Dstate.savepoints.dir
+ String savepointPath =
+ FlinkClient.extractDynamicPropertiesAsJava(application.getDynamicProperties())
+ .get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+
+ // Application conf configuration has the second priority. If it is a streampark|flinksql type
+ // task,
+ // see if Application conf is configured when the task is defined, if checkpoints are configured
+ // and enabled,
+ // read `state.savepoints.dir`
+ if (StringUtils.isBlank(savepointPath)) {
+ if (application.isStreamParkJob() || application.isFlinkSqlJob()) {
+ ApplicationConfig applicationConfig = configService.getEffective(application.getId());
+ if (applicationConfig != null) {
+ Map<String, String> map = applicationConfig.readConfig();
+ if (FlinkUtils.isCheckpointEnabled(map)) {
+ savepointPath = map.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+ }
+ }
+
+ // 3) If the savepoint is not obtained above, try to obtain the savepoint path according to the
+ // deployment type (remote|on yarn)
+ if (StringUtils.isBlank(savepointPath)) {
+ // 3.1) At the remote mode, request the flink webui interface to get the savepoint path
+ if (ExecutionMode.isRemoteMode(application.getExecutionMode())) {
+ FlinkCluster cluster = flinkClusterService.getById(application.getFlinkClusterId());
+ Utils.required(
+ cluster != null,
+ String.format(
+ "The clusterId=%s cannot be find, maybe the clusterId is wrong or "
+ + "the cluster has been deleted. Please contact the Admin.",
+ application.getFlinkClusterId()));
+ Map<String, String> config = cluster.getFlinkConfig();
+ if (!config.isEmpty()) {
+ savepointPath = config.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ } else {
+ // 3.2) At the yarn or k8s mode, then read the savepoint in flink-conf.yml in the bound
+ // flink
+ FlinkEnv flinkEnv = flinkEnvService.getById(application.getVersionId());
+ savepointPath =
+ flinkEnv.convertFlinkYamlAsMap().get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+
+ return savepointPath;
+ }
+
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application application = applicationService.getById(appId);
+
+ FlinkRESTAPIWatcher.addSavepoint(application.getId());
+
+ application.setOptionState(OptionState.SAVEPOINTING.getValue());
+ application.setOptionTime(new Date());
+ this.applicationService.updateById(application);
+
+ FlinkEnv flinkEnv = flinkEnvService.getById(application.getVersionId());
+
+ // infer savepoint
+ String customSavepoint = this.getFinalSavepointDir(savepointPath, application);
+
+ FlinkCluster cluster = flinkClusterService.getById(application.getFlinkClusterId());
+ String clusterId = getClusterId(application, cluster);
+
+ Map<String, Object> properties = this.tryGetRestProps(application, cluster);
+
+ TriggerSavepointRequest request =
+ new TriggerSavepointRequest(
+ flinkEnv.getFlinkVersion(),
+ ExecutionMode.of(application.getExecutionMode()),
+ clusterId,
+ application.getJobId(),
+ customSavepoint,
+ application.getK8sNamespace(),
+ properties);
+
+ CompletableFuture<SavepointResponse> savepointFuture =
+ CompletableFuture.supplyAsync(() -> FlinkClient.triggerSavepoint(request), executorService);
+
+ handleSavepointResponseFuture(application, savepointFuture);
+ }
+
+ private void handleSavepointResponseFuture(
+ Application application, CompletableFuture<SavepointResponse> savepointFuture) {
+ CompletableFutureUtils.runTimeout(
+ savepointFuture,
+ 10L,
+ TimeUnit.MINUTES,
+ savepointResponse -> {
+ if (savepointResponse != null && savepointResponse.savePointDir() != null) {
+ String savePointDir = savepointResponse.savePointDir();
+ log.info("Request savepoint successful, savepointDir: {}", savePointDir);
+ }
+ },
+ e -> {
+ log.error("Trigger savepoint for flink job failed.", e);
+ ApplicationLog log = new ApplicationLog();
+ log.setAppId(application.getId());
+ log.setYarnAppId(application.getClusterId());
+ log.setOptionTime(new Date());
+ String exception = Utils.stringifyException(e);
+ log.setException(exception);
+ if (!(e instanceof TimeoutException)) {
+ log.setSuccess(false);
+ }
+ applicationLogService.save(log);
+ })
+ .whenComplete(
+ (t, e) -> {
+ application.setOptionState(OptionState.NONE.getValue());
+ application.setOptionTime(new Date());
+ applicationService.update(application);
+ });
+ }
+
+ private String getFinalSavepointDir(@Nullable String savepointPath, Application application) {
+ String result = savepointPath;
+ if (StringUtils.isEmpty(savepointPath)) {
+ try {
+ result = this.getSavePointPath(application);
+ } catch (Exception e) {
+ throw new ApiAlertException(
+ "Error in getting savepoint path for triggering savepoint for app "
+ + application.getId(),
+ e);
+ }
+ }
+ return result;
+ }
+
+ @NotNull
+ private Map<String, Object> tryGetRestProps(Application application, FlinkCluster cluster) {
+ Map<String, Object> properties = new HashMap<>();
+
+ if (ExecutionMode.isRemoteMode(application.getExecutionModeEnum())) {
+ Utils.required(
+ cluster != null,
+ String.format(
+ "The clusterId=%s cannot be find, maybe the clusterId is wrong or "
+ + "the cluster has been deleted. Please contact the Admin.",
+ application.getFlinkClusterId()));
+ URI activeAddress = cluster.getRemoteURI();
+ properties.put(RestOptions.ADDRESS.key(), activeAddress.getHost());
+ properties.put(RestOptions.PORT.key(), activeAddress.getPort());
+ }
+ return properties;
+ }
+
+ private String getClusterId(Application application, FlinkCluster cluster) {
+ if (ExecutionMode.isKubernetesMode(application.getExecutionMode())) {
+ return application.getClusterId();
+ } else if (ExecutionMode.isYarnMode(application.getExecutionMode())) {
+ if (ExecutionMode.YARN_SESSION.equals(application.getExecutionModeEnum())) {
+ Utils.required(
Review Comment:
Utils.notNull(cluster, ...)
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -171,6 +218,182 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public String getSavePointPath(Application appParam) throws Exception {
+ Application application = applicationService.getById(appParam.getId());
+
+ // 1) properties have the highest priority, read the properties are set: -Dstate.savepoints.dir
+ String savepointPath =
+ FlinkClient.extractDynamicPropertiesAsJava(application.getDynamicProperties())
+ .get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+
+ // Application conf configuration has the second priority. If it is a streampark|flinksql type
+ // task,
+ // see if Application conf is configured when the task is defined, if checkpoints are configured
+ // and enabled,
+ // read `state.savepoints.dir`
+ if (StringUtils.isBlank(savepointPath)) {
+ if (application.isStreamParkJob() || application.isFlinkSqlJob()) {
+ ApplicationConfig applicationConfig = configService.getEffective(application.getId());
+ if (applicationConfig != null) {
+ Map<String, String> map = applicationConfig.readConfig();
+ if (FlinkUtils.isCheckpointEnabled(map)) {
+ savepointPath = map.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+ }
+ }
+
+ // 3) If the savepoint is not obtained above, try to obtain the savepoint path according to the
+ // deployment type (remote|on yarn)
+ if (StringUtils.isBlank(savepointPath)) {
+ // 3.1) At the remote mode, request the flink webui interface to get the savepoint path
+ if (ExecutionMode.isRemoteMode(application.getExecutionMode())) {
+ FlinkCluster cluster = flinkClusterService.getById(application.getFlinkClusterId());
+ Utils.required(
+ cluster != null,
+ String.format(
+ "The clusterId=%s cannot be find, maybe the clusterId is wrong or "
+ + "the cluster has been deleted. Please contact the Admin.",
+ application.getFlinkClusterId()));
+ Map<String, String> config = cluster.getFlinkConfig();
+ if (!config.isEmpty()) {
+ savepointPath = config.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ } else {
+ // 3.2) At the yarn or k8s mode, then read the savepoint in flink-conf.yml in the bound
+ // flink
+ FlinkEnv flinkEnv = flinkEnvService.getById(application.getVersionId());
+ savepointPath =
+ flinkEnv.convertFlinkYamlAsMap().get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+
+ return savepointPath;
+ }
+
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application application = applicationService.getById(appId);
+
+ FlinkRESTAPIWatcher.addSavepoint(application.getId());
+
+ application.setOptionState(OptionState.SAVEPOINTING.getValue());
+ application.setOptionTime(new Date());
+ this.applicationService.updateById(application);
Review Comment:
after application updated, need to be call: FlinkRESTAPIWatcher.init()
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -171,6 +218,182 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public String getSavePointPath(Application appParam) throws Exception {
+ Application application = applicationService.getById(appParam.getId());
+
+ // 1) properties have the highest priority, read the properties are set: -Dstate.savepoints.dir
+ String savepointPath =
+ FlinkClient.extractDynamicPropertiesAsJava(application.getDynamicProperties())
+ .get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+
+ // Application conf configuration has the second priority. If it is a streampark|flinksql type
+ // task,
+ // see if Application conf is configured when the task is defined, if checkpoints are configured
+ // and enabled,
+ // read `state.savepoints.dir`
+ if (StringUtils.isBlank(savepointPath)) {
+ if (application.isStreamParkJob() || application.isFlinkSqlJob()) {
+ ApplicationConfig applicationConfig = configService.getEffective(application.getId());
+ if (applicationConfig != null) {
+ Map<String, String> map = applicationConfig.readConfig();
+ if (FlinkUtils.isCheckpointEnabled(map)) {
+ savepointPath = map.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+ }
+ }
+
+ // 3) If the savepoint is not obtained above, try to obtain the savepoint path according to the
+ // deployment type (remote|on yarn)
+ if (StringUtils.isBlank(savepointPath)) {
+ // 3.1) At the remote mode, request the flink webui interface to get the savepoint path
+ if (ExecutionMode.isRemoteMode(application.getExecutionMode())) {
+ FlinkCluster cluster = flinkClusterService.getById(application.getFlinkClusterId());
+ Utils.required(
+ cluster != null,
+ String.format(
+ "The clusterId=%s cannot be find, maybe the clusterId is wrong or "
+ + "the cluster has been deleted. Please contact the Admin.",
+ application.getFlinkClusterId()));
+ Map<String, String> config = cluster.getFlinkConfig();
+ if (!config.isEmpty()) {
+ savepointPath = config.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ } else {
+ // 3.2) At the yarn or k8s mode, then read the savepoint in flink-conf.yml in the bound
+ // flink
+ FlinkEnv flinkEnv = flinkEnvService.getById(application.getVersionId());
+ savepointPath =
+ flinkEnv.convertFlinkYamlAsMap().get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+
+ return savepointPath;
+ }
+
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application application = applicationService.getById(appId);
+
+ FlinkRESTAPIWatcher.addSavepoint(application.getId());
+
+ application.setOptionState(OptionState.SAVEPOINTING.getValue());
+ application.setOptionTime(new Date());
+ this.applicationService.updateById(application);
+
+ FlinkEnv flinkEnv = flinkEnvService.getById(application.getVersionId());
+
+ // infer savepoint
+ String customSavepoint = this.getFinalSavepointDir(savepointPath, application);
+
+ FlinkCluster cluster = flinkClusterService.getById(application.getFlinkClusterId());
+ String clusterId = getClusterId(application, cluster);
+
+ Map<String, Object> properties = this.tryGetRestProps(application, cluster);
+
+ TriggerSavepointRequest request =
+ new TriggerSavepointRequest(
+ flinkEnv.getFlinkVersion(),
+ ExecutionMode.of(application.getExecutionMode()),
+ clusterId,
+ application.getJobId(),
+ customSavepoint,
+ application.getK8sNamespace(),
+ properties);
+
+ CompletableFuture<SavepointResponse> savepointFuture =
+ CompletableFuture.supplyAsync(() -> FlinkClient.triggerSavepoint(request), executorService);
+
+ handleSavepointResponseFuture(application, savepointFuture);
+ }
+
+ private void handleSavepointResponseFuture(
+ Application application, CompletableFuture<SavepointResponse> savepointFuture) {
+ CompletableFutureUtils.runTimeout(
+ savepointFuture,
+ 10L,
+ TimeUnit.MINUTES,
+ savepointResponse -> {
+ if (savepointResponse != null && savepointResponse.savePointDir() != null) {
+ String savePointDir = savepointResponse.savePointDir();
+ log.info("Request savepoint successful, savepointDir: {}", savePointDir);
+ }
+ },
+ e -> {
+ log.error("Trigger savepoint for flink job failed.", e);
+ ApplicationLog log = new ApplicationLog();
+ log.setAppId(application.getId());
+ log.setYarnAppId(application.getClusterId());
Review Comment:
yarn appid? It seems inaccurate, if execmode is yarn, it needs to be set, but k8s mode is not
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -171,6 +218,182 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public String getSavePointPath(Application appParam) throws Exception {
+ Application application = applicationService.getById(appParam.getId());
+
+ // 1) properties have the highest priority, read the properties are set: -Dstate.savepoints.dir
+ String savepointPath =
+ FlinkClient.extractDynamicPropertiesAsJava(application.getDynamicProperties())
+ .get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+
+ // Application conf configuration has the second priority. If it is a streampark|flinksql type
+ // task,
+ // see if Application conf is configured when the task is defined, if checkpoints are configured
+ // and enabled,
+ // read `state.savepoints.dir`
+ if (StringUtils.isBlank(savepointPath)) {
+ if (application.isStreamParkJob() || application.isFlinkSqlJob()) {
+ ApplicationConfig applicationConfig = configService.getEffective(application.getId());
+ if (applicationConfig != null) {
+ Map<String, String> map = applicationConfig.readConfig();
+ if (FlinkUtils.isCheckpointEnabled(map)) {
+ savepointPath = map.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+ }
+ }
+
+ // 3) If the savepoint is not obtained above, try to obtain the savepoint path according to the
+ // deployment type (remote|on yarn)
+ if (StringUtils.isBlank(savepointPath)) {
+ // 3.1) At the remote mode, request the flink webui interface to get the savepoint path
+ if (ExecutionMode.isRemoteMode(application.getExecutionMode())) {
+ FlinkCluster cluster = flinkClusterService.getById(application.getFlinkClusterId());
+ Utils.required(
+ cluster != null,
+ String.format(
+ "The clusterId=%s cannot be find, maybe the clusterId is wrong or "
+ + "the cluster has been deleted. Please contact the Admin.",
+ application.getFlinkClusterId()));
+ Map<String, String> config = cluster.getFlinkConfig();
+ if (!config.isEmpty()) {
+ savepointPath = config.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ } else {
+ // 3.2) At the yarn or k8s mode, then read the savepoint in flink-conf.yml in the bound
+ // flink
+ FlinkEnv flinkEnv = flinkEnvService.getById(application.getVersionId());
+ savepointPath =
+ flinkEnv.convertFlinkYamlAsMap().get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+
+ return savepointPath;
+ }
+
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application application = applicationService.getById(appId);
+
+ FlinkRESTAPIWatcher.addSavepoint(application.getId());
+
+ application.setOptionState(OptionState.SAVEPOINTING.getValue());
+ application.setOptionTime(new Date());
+ this.applicationService.updateById(application);
+
+ FlinkEnv flinkEnv = flinkEnvService.getById(application.getVersionId());
+
+ // infer savepoint
+ String customSavepoint = this.getFinalSavepointDir(savepointPath, application);
+
+ FlinkCluster cluster = flinkClusterService.getById(application.getFlinkClusterId());
+ String clusterId = getClusterId(application, cluster);
+
+ Map<String, Object> properties = this.tryGetRestProps(application, cluster);
+
+ TriggerSavepointRequest request =
+ new TriggerSavepointRequest(
+ flinkEnv.getFlinkVersion(),
+ ExecutionMode.of(application.getExecutionMode()),
+ clusterId,
+ application.getJobId(),
+ customSavepoint,
+ application.getK8sNamespace(),
+ properties);
+
+ CompletableFuture<SavepointResponse> savepointFuture =
+ CompletableFuture.supplyAsync(() -> FlinkClient.triggerSavepoint(request), executorService);
+
+ handleSavepointResponseFuture(application, savepointFuture);
+ }
+
+ private void handleSavepointResponseFuture(
+ Application application, CompletableFuture<SavepointResponse> savepointFuture) {
+ CompletableFutureUtils.runTimeout(
+ savepointFuture,
+ 10L,
+ TimeUnit.MINUTES,
+ savepointResponse -> {
+ if (savepointResponse != null && savepointResponse.savePointDir() != null) {
+ String savePointDir = savepointResponse.savePointDir();
+ log.info("Request savepoint successful, savepointDir: {}", savePointDir);
+ }
+ },
+ e -> {
+ log.error("Trigger savepoint for flink job failed.", e);
+ ApplicationLog log = new ApplicationLog();
+ log.setAppId(application.getId());
+ log.setYarnAppId(application.getClusterId());
+ log.setOptionTime(new Date());
+ String exception = Utils.stringifyException(e);
+ log.setException(exception);
+ if (!(e instanceof TimeoutException)) {
+ log.setSuccess(false);
+ }
+ applicationLogService.save(log);
+ })
+ .whenComplete(
+ (t, e) -> {
+ application.setOptionState(OptionState.NONE.getValue());
+ application.setOptionTime(new Date());
+ applicationService.update(application);
Review Comment:
after application update need to be call: FlinkRESTAPIWatcher.init()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] 1996fanrui commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1111222482
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/task/CheckpointProcessor.java:
##########
@@ -52,30 +66,32 @@ public class CheckpointProcessor {
@Autowired private SavePointService savePointService;
- public void process(Long appId, CheckPoints checkPoints) {
- CheckPoints.Latest latest = checkPoints.getLatest();
- if (latest == null || latest.getCompleted() == null) {
- return;
- }
- CheckPoints.CheckPoint checkPoint = latest.getCompleted();
- Application application = applicationService.getById(appId);
+ public void process(Application application, @Nonnull CheckPoints checkPoints) {
+ checkPoints.getLatestCheckpoint().forEach(checkPoint -> process(application, checkPoint));
+ }
+
+ private void process(Application application, @Nonnull CheckPoints.CheckPoint checkPoint) {
+ String jobID = application.getJobId();
+ Long appId = application.getId();
CheckPointStatus status = checkPoint.getCheckPointStatus();
if (CheckPointStatus.COMPLETED.equals(status)) {
- String cacheId = appId + "_" + application.getJobId();
- Long latestId =
- checkPointCache.get(
- cacheId,
- key -> {
- SavePoint savePoint = savePointService.getLatest(appId);
- return Optional.ofNullable(savePoint).map(SavePoint::getChkId).orElse(null);
- });
-
- if (latestId == null || latestId < checkPoint.getId()) {
- saveSavepoint(checkPoint, application);
+ String cacheId = getCacheIdForCheckpoint(appId, jobID);
+ Long latestChkId = getLatestCheckpointedId(appId, cacheId);
+ if (shouldStoreAsSavepoint(appId, jobID, checkPoint)) {
+ savepointedCache.put(
+ getCacheIdForSavepoint(appId, jobID, checkPoint.getId()), DEFAULT_FLAG_BYTE);
+ saveSavepoint(
+ checkPoint,
+ application.getId(),
+ latestChkId == null || checkPoint.getId() > latestChkId);
+ return;
+ }
+ if (shouldStoreAsCheckpoint(checkPoint, latestChkId)) {
Review Comment:
If so, why does savepoint use the latestChkId?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] wolfboys commented on a diff in pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#discussion_r1112021616
##########
streampark-console/streampark-console-service/src/main/java/org/apache/streampark/console/core/service/impl/SavePointServiceImpl.java:
##########
@@ -171,6 +220,184 @@ public SavePoint getLatest(Long id) {
return this.getOne(queryWrapper);
}
+ @Override
+ public String getSavePointPath(Application appParam) throws Exception {
+ Application application = applicationService.getById(appParam.getId());
+
+ // 1) properties have the highest priority, read the properties are set: -Dstate.savepoints.dir
+ String savepointPath =
+ FlinkClient.extractDynamicPropertiesAsJava(application.getDynamicProperties())
+ .get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+
+ // Application conf configuration has the second priority. If it is a streampark|flinksql type
+ // task,
+ // see if Application conf is configured when the task is defined, if checkpoints are configured
+ // and enabled,
+ // read `state.savepoints.dir`
+ if (StringUtils.isBlank(savepointPath)) {
+ if (application.isStreamParkJob() || application.isFlinkSqlJob()) {
+ ApplicationConfig applicationConfig = configService.getEffective(application.getId());
+ if (applicationConfig != null) {
+ Map<String, String> map = applicationConfig.readConfig();
+ if (FlinkUtils.isCheckpointEnabled(map)) {
+ savepointPath = map.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+ }
+ }
+
+ // 3) If the savepoint is not obtained above, try to obtain the savepoint path according to the
+ // deployment type (remote|on yarn)
+ if (StringUtils.isBlank(savepointPath)) {
+ // 3.1) At the remote mode, request the flink webui interface to get the savepoint path
+ if (ExecutionMode.isRemoteMode(application.getExecutionMode())) {
+ FlinkCluster cluster = flinkClusterService.getById(application.getFlinkClusterId());
+ Utils.notNull(
+ cluster,
+ String.format(
+ "The clusterId=%s cannot be find, maybe the clusterId is wrong or "
+ + "the cluster has been deleted. Please contact the Admin.",
+ application.getFlinkClusterId()));
+ Map<String, String> config = cluster.getFlinkConfig();
+ if (!config.isEmpty()) {
+ savepointPath = config.get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ } else {
+ // 3.2) At the yarn or k8s mode, then read the savepoint in flink-conf.yml in the bound
+ // flink
+ FlinkEnv flinkEnv = flinkEnvService.getById(application.getVersionId());
+ savepointPath =
+ flinkEnv.convertFlinkYamlAsMap().get(CheckpointingOptions.SAVEPOINT_DIRECTORY.key());
+ }
+ }
+
+ return savepointPath;
+ }
+
+ @Override
+ public void trigger(Long appId, @Nullable String savepointPath) {
+ log.info("Start to trigger savepoint for app {}", appId);
+ Application application = applicationService.getById(appId);
+
+ FlinkRESTAPIWatcher.addSavepoint(application.getId());
+
+ application.setOptionState(OptionState.SAVEPOINTING.getValue());
+ application.setOptionTime(new Date());
+ this.applicationService.updateById(application);
+ flinkRESTAPIWatcher.init();
+
+ FlinkEnv flinkEnv = flinkEnvService.getById(application.getVersionId());
+
+ // infer savepoint
+ String customSavepoint = this.getFinalSavepointDir(savepointPath, application);
+
+ FlinkCluster cluster = flinkClusterService.getById(application.getFlinkClusterId());
+ String clusterId = getClusterId(application, cluster);
+
+ Map<String, Object> properties = this.tryGetRestProps(application, cluster);
+
+ TriggerSavepointRequest request =
+ new TriggerSavepointRequest(
+ flinkEnv.getFlinkVersion(),
+ ExecutionMode.of(application.getExecutionMode()),
Review Comment:
code can be improved:
`application.getExecutionModeEnum()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-streampark] RocMarshal commented on pull request #2268: [Feature][Issue-2192] Support trigger savepoint manually.
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #2268:
URL: https://github.com/apache/incubator-streampark/pull/2268#issuecomment-1441926829
Thank all of you for the review & help.👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org