You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "fangwentong (JIRA)" <ji...@apache.org> on 2017/11/23 05:10:00 UTC
[jira] [Created] (FILEUPLOAD-288) Linux dentry cache memory leak
due to inappropriate temporary file detection
fangwentong created FILEUPLOAD-288:
--------------------------------------
Summary: Linux dentry cache memory leak due to inappropriate temporary file detection
Key: FILEUPLOAD-288
URL: https://issues.apache.org/jira/browse/FILEUPLOAD-288
Project: Commons FileUpload
Issue Type: Improvement
Environment: OS version: CentOS release 6.7 (Final)
Kenel: 2.6.32-573.26.1.el6.x86_64
JVM: Oracle 1.8.0_102-b14
Tomcat: 8.0.33.0
commons-fileupload : 1.3.3
Reporter: fangwentong
We use commons-fileupload:1.3.3 to handle multipart request, param {{file-size-threshold}} was set to 200K. Our API handle multipart request in high QPS, the size of request body is about 20KB, which means those parts was saved in memory, not written to disk temporary file.
But, in our production environment, size of dentry cache grows up in high speed. here are some analysis:
in {{strace}} result, plenty of {{stat}} system call invoked for tmp files which not exists.
{noformat}
[pid 111310] <... stat resumed> {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0
[pid 57844] <... stat resumed> 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624923.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624922.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624921.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624920.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624919.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624918.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 69798] stat("/tmp", <unfinished ...>
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624917.tmp", <unfinished ...>
[pid 69798] <... stat resumed> {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0
[pid 57844] <... stat resumed> 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624916.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624915.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624914.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624913.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624912.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624911.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624910.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624909.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624908.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624907.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624906.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624905.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624904.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624903.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624902.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624901.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624900.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624899.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624898.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624897.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624896.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624895.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624894.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624893.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624892.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624891.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624890.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624889.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624888.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624887.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624886.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624885.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624884.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 78454] --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=57810, si_uid=605} ---{noformat}
in {{slabtop}} result , dentry cache taken up 22GB+ memory.
{noformat}
Active / Total Objects (% used) : 233715492 / 234153507 (99.8%)
Active / Total Slabs (% used) : 7858579 / 7858606 (100.0%)
Active / Total Caches (% used) : 108 / 200 (54.0%)
Active / Total Size (% used) : 29322726.34K / 29380106.43K (99.8%)
Minimum / Average / Maximum Object : 0.02K / 0.12K / 4096.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
116095303 116095295 26% 0.06K 1967717 59 7870868K size-64
116090260 116090260 26% 0.19K 5804513 20 23218052K dentry
{noformat}
finally, I think the root cause is：in temporary file cleanup policy defined at {{DiskFileItem#finalize}}, call {{outputFile.exists()}} for every File, even if the part not write to disk.
{code:java}
@Override
protected void finalize() {
if (dfos == null) {
return;
}
File outputFile = dfos.getFile();
if (outputFile != null && outputFile.exists()) {
outputFile.delete();
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)