You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "niumy0701 (via GitHub)" <gi...@apache.org> on 2025/11/25 08:11:57 UTC
[PR] [Improvement-17725][TaskPlugin] SagemakerTask resource leak repair [dolphinscheduler]
niumy0701 opened a new pull request, #17726:
URL: https://github.com/apache/dolphinscheduler/pull/17726
Please refer to the details of main issue https://github.com/apache/dolphinscheduler/issues/14877
close sub issue https://github.com/apache/dolphinscheduler/issues/17725
<!--Thanks very much for contributing to Apache DolphinScheduler, we are happy that you want to help us improve DolphinScheduler! -->
## Purpose of the pull request
<!--(For example: This pull request adds checkstyle plugin).-->
## Brief change log
<!--*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
-->
## Verify this pull request
<!--*(Please pick either of the following options)*-->
This pull request is code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
<!--*(example:)*
- *Added dolphinscheduler-dao tests for end-to-end.*
- *Added CronUtilsTest to verify the change.*
- *Manually verified the change by testing locally.* -->
(or)
## Pull Request Notice
[Pull Request Notice](https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)
If your pull request contains incompatible change, you should also add it to `docs/docs/en/guide/upgrade/incompatible.md`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17725][TaskPlugin] SagemakerTask resource leak repair [dolphinscheduler]
Posted by "niumy0701 (via GitHub)" <gi...@apache.org>.
niumy0701 commented on code in PR #17726:
URL: https://github.com/apache/dolphinscheduler/pull/17726#discussion_r2562560802
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/test/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerTaskTest.java:
##########
@@ -139,4 +145,33 @@ private String buildParameters() {
return JSONUtils.toJsonString(parameters);
}
+
+ @Test
+ public void testTrackApplicationStatus_InitPipelineIdThrowsException() throws Exception {
+ doThrow(new TaskException("sagemaker applicationID is null")).when(sagemakerTask).initPipelineId();
+
+ TaskException exception = assertThrows(TaskException.class, () -> {
+ sagemakerTask.trackApplicationStatus();
+ });
+
+ assertEquals("sagemaker applicationID is null", exception.getMessage());
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17725][TaskPlugin] SagemakerTask resource leak repair [dolphinscheduler]
Posted by "niumy0701 (via GitHub)" <gi...@apache.org>.
niumy0701 commented on code in PR #17726:
URL: https://github.com/apache/dolphinscheduler/pull/17726#discussion_r2562515280
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/main/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerTask.java:
##########
@@ -126,26 +126,36 @@ public void submitApplication() throws TaskException {
@Override
public void cancelApplication() {
- initPipelineId();
try {
+ initPipelineId();
// stop pipeline
utils.stopPipelineExecution(client, pipelineId);
} catch (Exception e) {
throw new TaskException("cancel application error", e);
+ } finally {
+ // shutdown client
+ client.shutdown();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17725][TaskPlugin] SagemakerTask resource leak repair [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun closed pull request #17726: [Improvement-17725][TaskPlugin] SagemakerTask resource leak repair
URL: https://github.com/apache/dolphinscheduler/pull/17726
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17725][TaskPlugin] SagemakerTask resource leak repair [dolphinscheduler]
Posted by "niumy0701 (via GitHub)" <gi...@apache.org>.
niumy0701 commented on code in PR #17726:
URL: https://github.com/apache/dolphinscheduler/pull/17726#discussion_r2562424805
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/main/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerTask.java:
##########
@@ -126,26 +126,36 @@ public void submitApplication() throws TaskException {
@Override
public void cancelApplication() {
- initPipelineId();
try {
+ initPipelineId();
// stop pipeline
utils.stopPipelineExecution(client, pipelineId);
} catch (Exception e) {
throw new TaskException("cancel application error", e);
+ } finally {
+ // shutdown client
+ client.shutdown();
}
}
@Override
public void trackApplicationStatus() throws TaskException {
- initPipelineId();
- // Keep checking the health status
- exitStatusCode = utils.checkPipelineExecutionStatus(client, pipelineId);
+ try {
+ initPipelineId();
+ // Keep checking the health status
+ exitStatusCode = utils.checkPipelineExecutionStatus(client, pipelineId);
+ } catch (Exception e) {
+ throw new TaskException(e.getMessage(), e);
+ } finally {
+ // shutdown client
+ client.shutdown();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17725][TaskPlugin] SagemakerTask resource leak repair [dolphinscheduler]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17726:
URL: https://github.com/apache/dolphinscheduler/pull/17726#discussion_r2560321052
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/main/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerTask.java:
##########
@@ -126,26 +126,36 @@ public void submitApplication() throws TaskException {
@Override
public void cancelApplication() {
- initPipelineId();
try {
+ initPipelineId();
// stop pipeline
utils.stopPipelineExecution(client, pipelineId);
} catch (Exception e) {
throw new TaskException("cancel application error", e);
+ } finally {
+ // shutdown client
+ client.shutdown();
}
}
@Override
public void trackApplicationStatus() throws TaskException {
- initPipelineId();
- // Keep checking the health status
- exitStatusCode = utils.checkPipelineExecutionStatus(client, pipelineId);
+ try {
+ initPipelineId();
+ // Keep checking the health status
+ exitStatusCode = utils.checkPipelineExecutionStatus(client, pipelineId);
+ } catch (Exception e) {
Review Comment:
The exception handling in `trackApplicationStatus()` catches all exceptions and re-throws them as TaskException, but this loses the original exception type information. For TaskException specifically, this creates unnecessary exception wrapping (a TaskException wrapped in another TaskException). Consider checking if the caught exception is already a TaskException and re-throwing it directly, only wrapping non-TaskException types.
```suggestion
} catch (Exception e) {
if (e instanceof TaskException) {
throw (TaskException) e;
}
```
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/main/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerTask.java:
##########
@@ -126,26 +126,36 @@ public void submitApplication() throws TaskException {
@Override
public void cancelApplication() {
- initPipelineId();
try {
+ initPipelineId();
// stop pipeline
utils.stopPipelineExecution(client, pipelineId);
} catch (Exception e) {
throw new TaskException("cancel application error", e);
+ } finally {
+ // shutdown client
+ client.shutdown();
Review Comment:
The `client.shutdown()` call in the finally block could throw a NullPointerException if the client is null (e.g., if `init()` was never called or failed before creating the client). Consider adding a null check: `if (client != null) { client.shutdown(); }` to prevent exceptions during cleanup.
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/main/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerTask.java:
##########
@@ -126,26 +126,36 @@ public void submitApplication() throws TaskException {
@Override
public void cancelApplication() {
- initPipelineId();
try {
+ initPipelineId();
// stop pipeline
utils.stopPipelineExecution(client, pipelineId);
} catch (Exception e) {
throw new TaskException("cancel application error", e);
+ } finally {
+ // shutdown client
+ client.shutdown();
}
}
@Override
public void trackApplicationStatus() throws TaskException {
- initPipelineId();
- // Keep checking the health status
- exitStatusCode = utils.checkPipelineExecutionStatus(client, pipelineId);
+ try {
+ initPipelineId();
+ // Keep checking the health status
+ exitStatusCode = utils.checkPipelineExecutionStatus(client, pipelineId);
+ } catch (Exception e) {
+ throw new TaskException(e.getMessage(), e);
+ } finally {
+ // shutdown client
+ client.shutdown();
+ }
}
/**
* init sagemaker applicationId if null
*/
- private void initPipelineId() {
+ public void initPipelineId() {
Review Comment:
[nitpick] The `initPipelineId()` method visibility was changed from `private` to `public` solely to enable mocking in tests. This breaks encapsulation and exposes internal implementation details. Consider using package-private visibility instead (no access modifier) which would allow test access while maintaining better encapsulation, or use reflection-based mocking approaches that don't require changing method visibility.
```suggestion
void initPipelineId() {
```
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/main/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerTask.java:
##########
@@ -126,26 +126,36 @@ public void submitApplication() throws TaskException {
@Override
public void cancelApplication() {
- initPipelineId();
try {
+ initPipelineId();
// stop pipeline
utils.stopPipelineExecution(client, pipelineId);
} catch (Exception e) {
throw new TaskException("cancel application error", e);
+ } finally {
+ // shutdown client
+ client.shutdown();
}
}
@Override
public void trackApplicationStatus() throws TaskException {
- initPipelineId();
- // Keep checking the health status
- exitStatusCode = utils.checkPipelineExecutionStatus(client, pipelineId);
+ try {
+ initPipelineId();
+ // Keep checking the health status
+ exitStatusCode = utils.checkPipelineExecutionStatus(client, pipelineId);
+ } catch (Exception e) {
+ throw new TaskException(e.getMessage(), e);
+ } finally {
+ // shutdown client
+ client.shutdown();
Review Comment:
The `client.shutdown()` call in the finally block could throw a NullPointerException if the client is null (e.g., if `init()` was never called or failed before creating the client). Consider adding a null check: `if (client != null) { client.shutdown(); }` to prevent exceptions during cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17725][TaskPlugin] SagemakerTask resource leak repair [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on PR #17726:
URL: https://github.com/apache/dolphinscheduler/pull/17726#issuecomment-3630725123
Please provide relevant verification before and after the repair.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17725][TaskPlugin] SagemakerTask resource leak repair [dolphinscheduler]
Posted by "niumy0701 (via GitHub)" <gi...@apache.org>.
niumy0701 commented on code in PR #17726:
URL: https://github.com/apache/dolphinscheduler/pull/17726#discussion_r2562423345
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/main/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerTask.java:
##########
@@ -126,26 +126,36 @@ public void submitApplication() throws TaskException {
@Override
public void cancelApplication() {
- initPipelineId();
try {
+ initPipelineId();
// stop pipeline
utils.stopPipelineExecution(client, pipelineId);
} catch (Exception e) {
throw new TaskException("cancel application error", e);
+ } finally {
+ // shutdown client
+ client.shutdown();
}
}
@Override
public void trackApplicationStatus() throws TaskException {
- initPipelineId();
- // Keep checking the health status
- exitStatusCode = utils.checkPipelineExecutionStatus(client, pipelineId);
+ try {
+ initPipelineId();
+ // Keep checking the health status
+ exitStatusCode = utils.checkPipelineExecutionStatus(client, pipelineId);
+ } catch (Exception e) {
+ throw new TaskException(e.getMessage(), e);
+ } finally {
+ // shutdown client
+ client.shutdown();
+ }
}
/**
* init sagemaker applicationId if null
*/
- private void initPipelineId() {
+ public void initPipelineId() {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17725][TaskPlugin] SagemakerTask resource leak repair [dolphinscheduler]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17726:
URL: https://github.com/apache/dolphinscheduler/pull/17726#discussion_r2562517483
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/test/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerTaskTest.java:
##########
@@ -139,4 +145,33 @@ private String buildParameters() {
return JSONUtils.toJsonString(parameters);
}
+
+ @Test
+ public void testTrackApplicationStatus_InitPipelineIdThrowsException() throws Exception {
+ doThrow(new TaskException("sagemaker applicationID is null")).when(sagemakerTask).initPipelineId();
+
+ TaskException exception = assertThrows(TaskException.class, () -> {
+ sagemakerTask.trackApplicationStatus();
+ });
+
+ assertEquals("sagemaker applicationID is null", exception.getMessage());
Review Comment:
The expected exception message doesn't match the actual implementation. The `trackApplicationStatus()` method wraps the exception with the message "SageMaker task track application error: " + original message. The assertion should expect "SageMaker task track application error: sagemaker applicationID is null" instead of just "sagemaker applicationID is null".
```suggestion
assertEquals("SageMaker task track application error: sagemaker applicationID is null", exception.getMessage());
```
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/test/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerTaskTest.java:
##########
@@ -139,4 +145,33 @@ private String buildParameters() {
return JSONUtils.toJsonString(parameters);
}
+
+ @Test
+ public void testTrackApplicationStatus_InitPipelineIdThrowsException() throws Exception {
+ doThrow(new TaskException("sagemaker applicationID is null")).when(sagemakerTask).initPipelineId();
+
+ TaskException exception = assertThrows(TaskException.class, () -> {
+ sagemakerTask.trackApplicationStatus();
+ });
+
+ assertEquals("sagemaker applicationID is null", exception.getMessage());
+ verify(client, times(1)).shutdown();
+ }
+
+ @Test
+ public void testCancelApplication_InitPipelineIdThrowsException() {
+ // Mock the behavior of initPipelineId to throw an exception
+ doThrow(new TaskException("sagemaker applicationID is null")).when(sagemakerTask).initPipelineId();
+
+ // Call the method under test and expect an exception
+ TaskException exception = assertThrows(TaskException.class, () -> {
+ sagemakerTask.cancelApplication();
+ });
+
+ // Verify the exception message
+ assertEquals("cancel application error", exception.getMessage());
Review Comment:
The expected exception message doesn't match the actual implementation. The `cancelApplication()` method wraps the exception with the message "SageMaker task cancel application error: " + original message. The assertion should expect "SageMaker task cancel application error: sagemaker applicationID is null" instead of "cancel application error".
```suggestion
assertEquals("SageMaker task cancel application error: sagemaker applicationID is null", exception.getMessage());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improvement-17725][TaskPlugin] SagemakerTask resource leak repair [dolphinscheduler]
Posted by "niumy0701 (via GitHub)" <gi...@apache.org>.
niumy0701 commented on code in PR #17726:
URL: https://github.com/apache/dolphinscheduler/pull/17726#discussion_r2562435411
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-sagemaker/src/main/java/org/apache/dolphinscheduler/plugin/task/sagemaker/SagemakerTask.java:
##########
@@ -126,26 +126,36 @@ public void submitApplication() throws TaskException {
@Override
public void cancelApplication() {
- initPipelineId();
try {
+ initPipelineId();
// stop pipeline
utils.stopPipelineExecution(client, pipelineId);
} catch (Exception e) {
throw new TaskException("cancel application error", e);
+ } finally {
+ // shutdown client
+ client.shutdown();
}
}
@Override
public void trackApplicationStatus() throws TaskException {
- initPipelineId();
- // Keep checking the health status
- exitStatusCode = utils.checkPipelineExecutionStatus(client, pipelineId);
+ try {
+ initPipelineId();
+ // Keep checking the health status
+ exitStatusCode = utils.checkPipelineExecutionStatus(client, pipelineId);
+ } catch (Exception e) {
Review Comment:
Unified handling of exceptions and printing of error messages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org