[GitHub] [flink] wsry opened a new pull request #9750: [FLINK-14139][rest]Fix potential memory leak problem of rest server.

[GitBox <gi...@apache.org>]

wsry opened a new pull request #9750: [FLINK-14139][rest]Fix potential memory leak problem of rest server.
URL: https://github.com/apache/flink/pull/9750
 
 
   ## What is the purpose of the change
   
   The purpose of this pr is to fix the potential memory leak problem of rest server when using session/standalone cluster. More specifically, Flink's rest server uses netty decoder for http request processing and file uploading. However io.netty.handler.codec.http.multipart.DiskAttribute and io.netty.handler.codec.http.multipart.DiskFileUpload class of netty would register some temp files, including post chunks and upload file chunks, to java.io.DeleteOnExitHook which has a potential of memory leak, because the registered file names will never be deleted before the cluster stops. This pr attempts to solve the problem by disabling the registration of temp files to java.io.DeleteOnExitHook and makes Flink's shutdown hook to take the responsibility of temp file clean up.
   
   ## Brief change log
   
     - *The deleteOnExitTemporaryFile filed of io.netty.handler.codec.http.multipart.DiskAttribute and io.netty.handler.codec.http.multipart.DiskFileUpload is set to false to disable netty to register the shutdown hook.*
     - *Set the baseDirectory of io.netty.handler.codec.http.multipart.DiskAttribute to the file upload directory (shares the same directory with file upload), which will be cleaned up by Flink's shutdown hook when jvm exits.*
     - *Add tests to verify that no temp files will be registered to java.io.DeleteOnExitHook after the fix.*
   
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
     - *Based on the existing FileUploadHandlerTest which will register temp file (both post chunks and uploaded file chunks) to java.io.DeleteOnExitHook, after file uploading, each test case verifies that no file is register to java.io.DeleteOnExitHook. Before the fix, the test will fail and after the fix, the test will pass.*
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes / **no**)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
     - The serializers: (yes / **no** / don't know)
     - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
     - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (yes / **no**)
     - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services