You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/04/16 14:07:45 UTC
[GitHub] [flink-kubernetes-operator] Aitozi opened a new pull request, #169: [FLINK-27269] Clean up the jar file after submitting the job
Aitozi opened a new pull request, #169:
URL: https://github.com/apache/flink-kubernetes-operator/pull/169
This pr is meant to clean up the jar file after submitting the job to avoid the jar files exploding after serval upgrade
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] Aitozi commented on a diff in pull request #169: [FLINK-27269] Clean up the jar file after submitting the job
Posted by GitBox <gi...@apache.org>.
Aitozi commented on code in PR #169:
URL: https://github.com/apache/flink-kubernetes-operator/pull/169#discussion_r851840308
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/FlinkService.java:
##########
@@ -234,6 +238,22 @@ private JarUploadResponseBody jarUpload(FlinkSessionJob sessionJob, Configuratio
}
}
+ private void jarDelete(Configuration conf, String jarId) {
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] Aitozi commented on a diff in pull request #169: [FLINK-27269] Clean up the jar file after submitting the job
Posted by GitBox <gi...@apache.org>.
Aitozi commented on code in PR #169:
URL: https://github.com/apache/flink-kubernetes-operator/pull/169#discussion_r851954087
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/FlinkService.java:
##########
@@ -234,6 +238,22 @@ private JarUploadResponseBody jarUpload(FlinkSessionJob sessionJob, Configuratio
}
}
+ private void deleteJar(Configuration conf, String jarId) {
+ try (RestClusterClient<String> clusterClient =
+ (RestClusterClient<String>) getClusterClient(conf)) {
+ JarDeleteHeaders headers = JarDeleteHeaders.getInstance();
+ JarDeleteMessageParameters parameters = headers.getUnresolvedMessageParameters();
+ parameters.jarIdPathParameter.resolve(jarId);
+ clusterClient
+ .sendRequest(headers, parameters, EmptyRequestBody.getInstance())
+ .get(
+ operatorConfiguration.getFlinkClientTimeout().toSeconds(),
+ TimeUnit.SECONDS);
+ } catch (Exception e) {
+ LOG.error("Failed to delete the jar.", e);
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] Aitozi commented on pull request #169: [FLINK-27269] Clean up the jar file after submitting the job
Posted by GitBox <gi...@apache.org>.
Aitozi commented on PR #169:
URL: https://github.com/apache/flink-kubernetes-operator/pull/169#issuecomment-1101129155
@wangyang0918 Thanks for pointing this, I will also take a look then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] wangyang0918 merged pull request #169: [FLINK-27269] Clean up the jar file after submitting the job
Posted by GitBox <gi...@apache.org>.
wangyang0918 merged PR #169:
URL: https://github.com/apache/flink-kubernetes-operator/pull/169
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] Aitozi commented on pull request #169: [FLINK-27269] Clean up the jar file after submitting the job
Posted by GitBox <gi...@apache.org>.
Aitozi commented on PR #169:
URL: https://github.com/apache/flink-kubernetes-operator/pull/169#issuecomment-1101189983
> Using `lsof | grep delete` in JobManager pod and do not find any leaked the jars after deletion. It seems that `userCodeClassLoader.close()` in `PackagedProgram#close()` has worked as expected.
>
> +1 for merging.
👍🏻 Nice way to verify, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #169: [FLINK-27269] Clean up the jar file after submitting the job
Posted by GitBox <gi...@apache.org>.
gyfora commented on code in PR #169:
URL: https://github.com/apache/flink-kubernetes-operator/pull/169#discussion_r851797971
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/FlinkService.java:
##########
@@ -234,6 +238,22 @@ private JarUploadResponseBody jarUpload(FlinkSessionJob sessionJob, Configuratio
}
}
+ private void jarDelete(Configuration conf, String jarId) {
Review Comment:
nit: could we please change the name to `deleteJar`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] wangyang0918 commented on pull request #169: [FLINK-27269] Clean up the jar file after submitting the job
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on PR #169:
URL: https://github.com/apache/flink-kubernetes-operator/pull/169#issuecomment-1101104349
I am not sure whether deleting the submitted jars could actually work, which means freeing up the disk spaces. Because the classes are still in use in a dedicated user classloader. See https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java#L144.
I will have a manual verification soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-kubernetes-operator] wangyang0918 commented on a diff in pull request #169: [FLINK-27269] Clean up the jar file after submitting the job
Posted by GitBox <gi...@apache.org>.
wangyang0918 commented on code in PR #169:
URL: https://github.com/apache/flink-kubernetes-operator/pull/169#discussion_r851946179
##########
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/FlinkService.java:
##########
@@ -234,6 +238,22 @@ private JarUploadResponseBody jarUpload(FlinkSessionJob sessionJob, Configuratio
}
}
+ private void deleteJar(Configuration conf, String jarId) {
+ try (RestClusterClient<String> clusterClient =
+ (RestClusterClient<String>) getClusterClient(conf)) {
+ JarDeleteHeaders headers = JarDeleteHeaders.getInstance();
+ JarDeleteMessageParameters parameters = headers.getUnresolvedMessageParameters();
+ parameters.jarIdPathParameter.resolve(jarId);
+ clusterClient
+ .sendRequest(headers, parameters, EmptyRequestBody.getInstance())
+ .get(
+ operatorConfiguration.getFlinkClientTimeout().toSeconds(),
+ TimeUnit.SECONDS);
+ } catch (Exception e) {
+ LOG.error("Failed to delete the jar.", e);
Review Comment:
We need to print the `jarId` when failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org