You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "niumy0701 (via GitHub)" <gi...@apache.org> on 2025/11/25 02:16:42 UTC
[PR] [DSIP-23][TaskPlugin] EmrTask resource leak repair [dolphinscheduler]
niumy0701 opened a new pull request, #17719:
URL: https://github.com/apache/dolphinscheduler/pull/17719
<!--Thanks very much for contributing to Apache DolphinScheduler, we are happy that you want to help us improve DolphinScheduler! -->
Please refer to the details #14877
## Purpose of the pull request
<!--(For example: This pull request adds checkstyle plugin).-->
## Brief change log
<!--*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
-->
## Verify this pull request
<!--*(Please pick either of the following options)*-->
This pull request is code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
<!--*(example:)*
- *Added dolphinscheduler-dao tests for end-to-end.*
- *Added CronUtilsTest to verify the change.*
- *Manually verified the change by testing locally.* -->
(or)
## Pull Request Notice
[Pull Request Notice](https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)
If your pull request contains incompatible change, you should also add it to `docs/docs/en/guide/upgrade/incompatible.md`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17724][TaskPlugin] EmrTask resource leak repair [dolphinscheduler]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17719:
URL: https://github.com/apache/dolphinscheduler/pull/17719#issuecomment-3625536108
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=17719) **Quality Gate failed**
Failed conditions
 [52.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=17719&metric=new_coverage&view=list) (required ≥ 60%)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=17719)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [DSIP-23][TaskPlugin] EmrTask resource leak repair [dolphinscheduler]
Posted by "niumy0701 (via GitHub)" <gi...@apache.org>.
niumy0701 commented on PR #17719:
URL: https://github.com/apache/dolphinscheduler/pull/17719#issuecomment-3574165933
> You should create an issue first and link to it. DSIP-23 is a main issue, and you need to create a sub issue. @niumy0701
done
sub issue is #17724
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17724][TaskPlugin] EmrTask resource leak repair [dolphinscheduler]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17719:
URL: https://github.com/apache/dolphinscheduler/pull/17719#issuecomment-3625541352
## [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=17719) **Quality Gate failed**
Failed conditions
 [52.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=17719&metric=new_coverage&view=list) (required ≥ 60%)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=17719)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17724][TaskPlugin] EmrTask resource leak repair [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on code in PR #17719:
URL: https://github.com/apache/dolphinscheduler/pull/17719#discussion_r2591174330
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTask.java:
##########
@@ -180,11 +180,23 @@ private ClusterStatus getClusterStatus() {
@Override
public void cancelApplication() throws TaskException {
- log.info("trying terminate job flow, taskId:{}, clusterId:{}", this.taskExecutionContext.getTaskInstanceId(),
- clusterId);
- TerminateJobFlowsRequest terminateJobFlowsRequest = new TerminateJobFlowsRequest().withJobFlowIds(clusterId);
- TerminateJobFlowsResult terminateJobFlowsResult = emrClient.terminateJobFlows(terminateJobFlowsRequest);
- log.info("the result of terminate job flow is:{}", terminateJobFlowsResult);
+ try {
+ log.info("trying terminate job flow, taskId:{}, clusterId:{}",
+ this.taskExecutionContext.getTaskInstanceId(),
+ clusterId);
+ TerminateJobFlowsRequest terminateJobFlowsRequest =
+ new TerminateJobFlowsRequest().withJobFlowIds(clusterId);
+ TerminateJobFlowsResult terminateJobFlowsResult = emrClient.terminateJobFlows(terminateJobFlowsRequest);
+ log.info("the result of terminate job flow is:{}", terminateJobFlowsResult);
+ } catch (Exception e) {
+ log.error("emr job flow task cancel error: {}", e.getMessage(), e);
+ throw new TaskException("emr job flow task cancel error: " + e.getMessage(), e);
+ } finally {
+ // shutdown emrclient
+ if (emrClient != null) {
+ emrClient.shutdown();
+ }
+ }
Review Comment:
Revert this change.
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTask.java:
##########
@@ -187,26 +187,37 @@ private StepStatus getStepStatus() {
@Override
public void cancelApplication() throws TaskException {
- log.info("trying cancel emr step, taskId:{}, clusterId:{}, stepId:{}",
- this.taskExecutionContext.getTaskInstanceId(), clusterId, stepId);
- CancelStepsRequest cancelStepsRequest = new CancelStepsRequest().withClusterId(clusterId).withStepIds(stepId);
- CancelStepsResult cancelStepsResult = emrClient.cancelSteps(cancelStepsRequest);
+ try {
+ log.info("trying cancel emr step, taskId:{}, clusterId:{}, stepId:{}",
+ this.taskExecutionContext.getTaskInstanceId(), clusterId, stepId);
+ CancelStepsRequest cancelStepsRequest =
+ new CancelStepsRequest().withClusterId(clusterId).withStepIds(stepId);
+ CancelStepsResult cancelStepsResult = emrClient.cancelSteps(cancelStepsRequest);
+
+ if (cancelStepsResult == null) {
+ throw new EmrTaskException("cancel emr step failed");
+ }
- if (cancelStepsResult == null) {
- throw new EmrTaskException("cancel emr step failed");
- }
+ CancelStepsInfo cancelEmrStepInfo = cancelStepsResult.getCancelStepsInfoList()
+ .stream()
+ .filter(cancelStepsInfo -> cancelStepsInfo.getStepId().equals(stepId))
+ .findFirst()
+ .orElseThrow(() -> new EmrTaskException("cancel emr step failed"));
- CancelStepsInfo cancelEmrStepInfo = cancelStepsResult.getCancelStepsInfoList()
- .stream()
- .filter(cancelStepsInfo -> cancelStepsInfo.getStepId().equals(stepId))
- .findFirst()
- .orElseThrow(() -> new EmrTaskException("cancel emr step failed"));
+ if (CancelStepsRequestStatus.FAILED.toString().equals(cancelEmrStepInfo.getStatus())) {
+ throw new EmrTaskException("cancel emr step failed, message:" + cancelEmrStepInfo.getReason());
+ }
- if (CancelStepsRequestStatus.FAILED.toString().equals(cancelEmrStepInfo.getStatus())) {
- throw new EmrTaskException("cancel emr step failed, message:" + cancelEmrStepInfo.getReason());
+ log.info("the result of cancel emr step is:{}", cancelStepsResult);
+ } catch (EmrTaskException | SdkBaseException e) {
+ log.error("cancel emr step failed", e);
+ throw new TaskException("cancel emr step failed", e);
+ } finally {
+ // shutdown emrclient
+ if (emrClient != null) {
+ emrClient.shutdown();
+ }
Review Comment:
Revert this change. The client should be closed at handle method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17724][TaskPlugin] EmrTask resource leak repair [dolphinscheduler]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17719:
URL: https://github.com/apache/dolphinscheduler/pull/17719#discussion_r2562517210
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java:
##########
@@ -198,4 +204,34 @@ private EmrParameters buildErrorEmrTaskParameters() {
return emrParameters;
}
+
+ @Test
+ public void cancelApplication_CancelStepsResultIsNull_ShouldThrowException() {
+ Mockito.when(emrClient.cancelSteps(any(CancelStepsRequest.class))).thenReturn(null);
+
+ TaskException exception = Assertions.assertThrows(TaskException.class, () -> {
+ emrAddStepsTask.cancelApplication();
+ });
+
+ verify(emrClient, times(1)).shutdown();
+ }
+
+ @Test
+ public void cancelApplication_CancelStepsRequestFails_ShouldThrowException() {
+ CancelStepsInfo cancelStepsInfo = new CancelStepsInfo()
+ .withStepId("step-123")
+ .withStatus(CancelStepsRequestStatus.FAILED.toString())
+ .withReason("Test failure");
+
+ CancelStepsResult cancelStepsResult = new CancelStepsResult()
+ .withCancelStepsInfoList(cancelStepsInfo);
+
+ Mockito.when(emrClient.cancelSteps(any(CancelStepsRequest.class))).thenReturn(cancelStepsResult);
+
+ TaskException exception = Assertions.assertThrows(TaskException.class, () -> {
+ emrAddStepsTask.cancelApplication();
+ });
Review Comment:
[nitpick] The `exception` variable is declared but never used. Consider either using it for additional assertions (e.g., checking the exception message) or removing the variable assignment:
```java
Assertions.assertThrows(TaskException.class, () -> {
emrAddStepsTask.cancelApplication();
});
```
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java:
##########
@@ -198,4 +204,34 @@ private EmrParameters buildErrorEmrTaskParameters() {
return emrParameters;
}
+
+ @Test
+ public void cancelApplication_CancelStepsResultIsNull_ShouldThrowException() {
+ Mockito.when(emrClient.cancelSteps(any(CancelStepsRequest.class))).thenReturn(null);
+
+ TaskException exception = Assertions.assertThrows(TaskException.class, () -> {
+ emrAddStepsTask.cancelApplication();
+ });
+
+ verify(emrClient, times(1)).shutdown();
+ }
+
+ @Test
+ public void cancelApplication_CancelStepsRequestFails_ShouldThrowException() {
+ CancelStepsInfo cancelStepsInfo = new CancelStepsInfo()
+ .withStepId("step-123")
+ .withStatus(CancelStepsRequestStatus.FAILED.toString())
+ .withReason("Test failure");
+
+ CancelStepsResult cancelStepsResult = new CancelStepsResult()
+ .withCancelStepsInfoList(cancelStepsInfo);
+
+ Mockito.when(emrClient.cancelSteps(any(CancelStepsRequest.class))).thenReturn(cancelStepsResult);
+
+ TaskException exception = Assertions.assertThrows(TaskException.class, () -> {
+ emrAddStepsTask.cancelApplication();
+ });
+
+ verify(emrClient, times(1)).shutdown();
+ }
Review Comment:
Test coverage gap: There's no test for successful cancellation in `EmrAddStepsTask`, while `EmrJobFlowTask` has `testCancelApplication_Success()`. Consider adding a similar test to verify that `shutdown()` is called in the success path:
```java
@Test
public void testCancelApplication_Success() throws TaskException {
CancelStepsInfo cancelStepsInfo = new CancelStepsInfo()
.withStepId("step-xx")
.withStatus(CancelStepsRequestStatus.SUBMITTED.toString());
CancelStepsResult cancelStepsResult = new CancelStepsResult()
.withCancelStepsInfoList(cancelStepsInfo);
Mockito.when(emrClient.cancelSteps(any(CancelStepsRequest.class))).thenReturn(cancelStepsResult);
Mockito.doNothing().when(emrClient).shutdown();
Assertions.assertDoesNotThrow(() -> emrAddStepsTask.cancelApplication());
verify(emrClient, times(1)).cancelSteps(any(CancelStepsRequest.class));
verify(emrClient, times(1)).shutdown();
}
```
Note: You'll need to initialize `stepId` to "step-xx" in the test setup or use reflection to set it.
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java:
##########
@@ -198,4 +204,34 @@ private EmrParameters buildErrorEmrTaskParameters() {
return emrParameters;
}
+
+ @Test
+ public void cancelApplication_CancelStepsResultIsNull_ShouldThrowException() {
+ Mockito.when(emrClient.cancelSteps(any(CancelStepsRequest.class))).thenReturn(null);
+
+ TaskException exception = Assertions.assertThrows(TaskException.class, () -> {
+ emrAddStepsTask.cancelApplication();
+ });
+
+ verify(emrClient, times(1)).shutdown();
+ }
+
+ @Test
+ public void cancelApplication_CancelStepsRequestFails_ShouldThrowException() {
+ CancelStepsInfo cancelStepsInfo = new CancelStepsInfo()
+ .withStepId("step-123")
+ .withStatus(CancelStepsRequestStatus.FAILED.toString())
+ .withReason("Test failure");
+
+ CancelStepsResult cancelStepsResult = new CancelStepsResult()
+ .withCancelStepsInfoList(cancelStepsInfo);
Review Comment:
Test issue: The test creates a `CancelStepsInfo` with stepId "step-123", but the `emrAddStepsTask.stepId` is never initialized in this test (it remains null). When `cancelApplication()` filters the cancel steps info list by stepId at line 206, it won't find a match because it's filtering by `null` instead of "step-123".
Consider initializing the stepId in the test setup to properly test the FAILED status scenario, or use a stepId that matches what the task instance would have.
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/test/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTaskTest.java:
##########
@@ -198,4 +204,34 @@ private EmrParameters buildErrorEmrTaskParameters() {
return emrParameters;
}
+
+ @Test
+ public void cancelApplication_CancelStepsResultIsNull_ShouldThrowException() {
+ Mockito.when(emrClient.cancelSteps(any(CancelStepsRequest.class))).thenReturn(null);
+
+ TaskException exception = Assertions.assertThrows(TaskException.class, () -> {
+ emrAddStepsTask.cancelApplication();
+ });
Review Comment:
[nitpick] The `exception` variable is declared but never used. Consider either using it for additional assertions (e.g., checking the exception message) or removing the variable assignment:
```java
Assertions.assertThrows(TaskException.class, () -> {
emrAddStepsTask.cancelApplication();
});
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17724][TaskPlugin] EmrTask resource leak repair [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun closed pull request #17719: [Improvement-17724][TaskPlugin] EmrTask resource leak repair
URL: https://github.com/apache/dolphinscheduler/pull/17719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17724][TaskPlugin] EmrTask resource leak repair [dolphinscheduler]
Posted by "niumy0701 (via GitHub)" <gi...@apache.org>.
niumy0701 commented on PR #17719:
URL: https://github.com/apache/dolphinscheduler/pull/17719#issuecomment-3591592213
> same [#17718 (review)](https://github.com/apache/dolphinscheduler/pull/17718#pullrequestreview-3518378791)
There should be no problem, the resources have already been closed in the finally block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17724][TaskPlugin] EmrTask resource leak repair [dolphinscheduler]
Posted by "niumy0701 (via GitHub)" <gi...@apache.org>.
niumy0701 commented on code in PR #17719:
URL: https://github.com/apache/dolphinscheduler/pull/17719#discussion_r2591217862
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTask.java:
##########
@@ -187,26 +187,37 @@ private StepStatus getStepStatus() {
@Override
public void cancelApplication() throws TaskException {
- log.info("trying cancel emr step, taskId:{}, clusterId:{}, stepId:{}",
- this.taskExecutionContext.getTaskInstanceId(), clusterId, stepId);
- CancelStepsRequest cancelStepsRequest = new CancelStepsRequest().withClusterId(clusterId).withStepIds(stepId);
- CancelStepsResult cancelStepsResult = emrClient.cancelSteps(cancelStepsRequest);
+ try {
+ log.info("trying cancel emr step, taskId:{}, clusterId:{}, stepId:{}",
+ this.taskExecutionContext.getTaskInstanceId(), clusterId, stepId);
+ CancelStepsRequest cancelStepsRequest =
+ new CancelStepsRequest().withClusterId(clusterId).withStepIds(stepId);
+ CancelStepsResult cancelStepsResult = emrClient.cancelSteps(cancelStepsRequest);
+
+ if (cancelStepsResult == null) {
+ throw new EmrTaskException("cancel emr step failed");
+ }
- if (cancelStepsResult == null) {
- throw new EmrTaskException("cancel emr step failed");
- }
+ CancelStepsInfo cancelEmrStepInfo = cancelStepsResult.getCancelStepsInfoList()
+ .stream()
+ .filter(cancelStepsInfo -> cancelStepsInfo.getStepId().equals(stepId))
+ .findFirst()
+ .orElseThrow(() -> new EmrTaskException("cancel emr step failed"));
- CancelStepsInfo cancelEmrStepInfo = cancelStepsResult.getCancelStepsInfoList()
- .stream()
- .filter(cancelStepsInfo -> cancelStepsInfo.getStepId().equals(stepId))
- .findFirst()
- .orElseThrow(() -> new EmrTaskException("cancel emr step failed"));
+ if (CancelStepsRequestStatus.FAILED.toString().equals(cancelEmrStepInfo.getStatus())) {
+ throw new EmrTaskException("cancel emr step failed, message:" + cancelEmrStepInfo.getReason());
+ }
- if (CancelStepsRequestStatus.FAILED.toString().equals(cancelEmrStepInfo.getStatus())) {
- throw new EmrTaskException("cancel emr step failed, message:" + cancelEmrStepInfo.getReason());
+ log.info("the result of cancel emr step is:{}", cancelStepsResult);
+ } catch (EmrTaskException | SdkBaseException e) {
+ log.error("cancel emr step failed", e);
+ throw new TaskException("cancel emr step failed", e);
+ } finally {
+ // shutdown emrclient
+ if (emrClient != null) {
+ emrClient.shutdown();
+ }
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17724][TaskPlugin] EmrTask resource leak repair [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on PR #17719:
URL: https://github.com/apache/dolphinscheduler/pull/17719#issuecomment-3630724727
Please provide relevant verification before and after the repair.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17724][TaskPlugin] EmrTask resource leak repair [dolphinscheduler]
Posted by "niumy0701 (via GitHub)" <gi...@apache.org>.
niumy0701 commented on code in PR #17719:
URL: https://github.com/apache/dolphinscheduler/pull/17719#discussion_r2591217862
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrAddStepsTask.java:
##########
@@ -187,26 +187,37 @@ private StepStatus getStepStatus() {
@Override
public void cancelApplication() throws TaskException {
- log.info("trying cancel emr step, taskId:{}, clusterId:{}, stepId:{}",
- this.taskExecutionContext.getTaskInstanceId(), clusterId, stepId);
- CancelStepsRequest cancelStepsRequest = new CancelStepsRequest().withClusterId(clusterId).withStepIds(stepId);
- CancelStepsResult cancelStepsResult = emrClient.cancelSteps(cancelStepsRequest);
+ try {
+ log.info("trying cancel emr step, taskId:{}, clusterId:{}, stepId:{}",
+ this.taskExecutionContext.getTaskInstanceId(), clusterId, stepId);
+ CancelStepsRequest cancelStepsRequest =
+ new CancelStepsRequest().withClusterId(clusterId).withStepIds(stepId);
+ CancelStepsResult cancelStepsResult = emrClient.cancelSteps(cancelStepsRequest);
+
+ if (cancelStepsResult == null) {
+ throw new EmrTaskException("cancel emr step failed");
+ }
- if (cancelStepsResult == null) {
- throw new EmrTaskException("cancel emr step failed");
- }
+ CancelStepsInfo cancelEmrStepInfo = cancelStepsResult.getCancelStepsInfoList()
+ .stream()
+ .filter(cancelStepsInfo -> cancelStepsInfo.getStepId().equals(stepId))
+ .findFirst()
+ .orElseThrow(() -> new EmrTaskException("cancel emr step failed"));
- CancelStepsInfo cancelEmrStepInfo = cancelStepsResult.getCancelStepsInfoList()
- .stream()
- .filter(cancelStepsInfo -> cancelStepsInfo.getStepId().equals(stepId))
- .findFirst()
- .orElseThrow(() -> new EmrTaskException("cancel emr step failed"));
+ if (CancelStepsRequestStatus.FAILED.toString().equals(cancelEmrStepInfo.getStatus())) {
+ throw new EmrTaskException("cancel emr step failed, message:" + cancelEmrStepInfo.getReason());
+ }
- if (CancelStepsRequestStatus.FAILED.toString().equals(cancelEmrStepInfo.getStatus())) {
- throw new EmrTaskException("cancel emr step failed, message:" + cancelEmrStepInfo.getReason());
+ log.info("the result of cancel emr step is:{}", cancelStepsResult);
+ } catch (EmrTaskException | SdkBaseException e) {
+ log.error("cancel emr step failed", e);
+ throw new TaskException("cancel emr step failed", e);
+ } finally {
+ // shutdown emrclient
+ if (emrClient != null) {
+ emrClient.shutdown();
+ }
Review Comment:
it need to close resources when canceling tasks, right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17724][TaskPlugin] EmrTask resource leak repair [dolphinscheduler]
Posted by "niumy0701 (via GitHub)" <gi...@apache.org>.
niumy0701 commented on code in PR #17719:
URL: https://github.com/apache/dolphinscheduler/pull/17719#discussion_r2591249226
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTask.java:
##########
@@ -180,11 +180,23 @@ private ClusterStatus getClusterStatus() {
@Override
public void cancelApplication() throws TaskException {
- log.info("trying terminate job flow, taskId:{}, clusterId:{}", this.taskExecutionContext.getTaskInstanceId(),
- clusterId);
- TerminateJobFlowsRequest terminateJobFlowsRequest = new TerminateJobFlowsRequest().withJobFlowIds(clusterId);
- TerminateJobFlowsResult terminateJobFlowsResult = emrClient.terminateJobFlows(terminateJobFlowsRequest);
- log.info("the result of terminate job flow is:{}", terminateJobFlowsResult);
+ try {
+ log.info("trying terminate job flow, taskId:{}, clusterId:{}",
+ this.taskExecutionContext.getTaskInstanceId(),
+ clusterId);
+ TerminateJobFlowsRequest terminateJobFlowsRequest =
+ new TerminateJobFlowsRequest().withJobFlowIds(clusterId);
+ TerminateJobFlowsResult terminateJobFlowsResult = emrClient.terminateJobFlows(terminateJobFlowsRequest);
+ log.info("the result of terminate job flow is:{}", terminateJobFlowsResult);
+ } catch (Exception e) {
+ log.error("emr job flow task cancel error: {}", e.getMessage(), e);
+ throw new TaskException("emr job flow task cancel error: " + e.getMessage(), e);
+ } finally {
+ // shutdown emrclient
+ if (emrClient != null) {
+ emrClient.shutdown();
+ }
+ }
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17724][TaskPlugin] EmrTask resource leak repair [dolphinscheduler]
Posted by "niumy0701 (via GitHub)" <gi...@apache.org>.
niumy0701 commented on code in PR #17719:
URL: https://github.com/apache/dolphinscheduler/pull/17719#discussion_r2591249226
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-emr/src/main/java/org/apache/dolphinscheduler/plugin/task/emr/EmrJobFlowTask.java:
##########
@@ -180,11 +180,23 @@ private ClusterStatus getClusterStatus() {
@Override
public void cancelApplication() throws TaskException {
- log.info("trying terminate job flow, taskId:{}, clusterId:{}", this.taskExecutionContext.getTaskInstanceId(),
- clusterId);
- TerminateJobFlowsRequest terminateJobFlowsRequest = new TerminateJobFlowsRequest().withJobFlowIds(clusterId);
- TerminateJobFlowsResult terminateJobFlowsResult = emrClient.terminateJobFlows(terminateJobFlowsRequest);
- log.info("the result of terminate job flow is:{}", terminateJobFlowsResult);
+ try {
+ log.info("trying terminate job flow, taskId:{}, clusterId:{}",
+ this.taskExecutionContext.getTaskInstanceId(),
+ clusterId);
+ TerminateJobFlowsRequest terminateJobFlowsRequest =
+ new TerminateJobFlowsRequest().withJobFlowIds(clusterId);
+ TerminateJobFlowsResult terminateJobFlowsResult = emrClient.terminateJobFlows(terminateJobFlowsRequest);
+ log.info("the result of terminate job flow is:{}", terminateJobFlowsResult);
+ } catch (Exception e) {
+ log.error("emr job flow task cancel error: {}", e.getMessage(), e);
+ throw new TaskException("emr job flow task cancel error: " + e.getMessage(), e);
+ } finally {
+ // shutdown emrclient
+ if (emrClient != null) {
+ emrClient.shutdown();
+ }
+ }
Review Comment:
it need to close resources when canceling tasks, right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org