You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/04/19 02:45:01 UTC
[GitHub] [incubator-druid] kaka11chen opened a new issue #7508: Druid
historical node running caused os kernel creating too many kernel's dentry
cache slab objects.
kaka11chen opened a new issue #7508: Druid historical node running caused os kernel creating too many kernel's dentry cache slab objects.
URL: https://github.com/apache/incubator-druid/issues/7508
Druid historical node running caused os kernel creating too many kernel's dentry cache slab objects.
### Affected Version
all versions
### Description
Druid historical node running caused os kernel created too many kernel's dentry cache slab objects, run out of memory, and caused performance issue(Use 'sar -B' and found majfit/s high).
#### OS Kernel version:
2.6.32-754.9.1.el6.x86_64
#### meminfo: Slab objects occupy 65G, most is dentry.
$ cat /proc/meminfo
MemTotal: 198340116 kB
MemFree: 3411764 kB
Buffers: 329324 kB
Cached: 50916944 kB
SwapCached: 0 kB
Active: 81630316 kB
Inactive: 44296244 kB
Active(anon): 74680324 kB
Inactive(anon): 148 kB
Active(file): 6949992 kB
Inactive(file): 44296096 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 136 kB
Writeback: 0 kB
AnonPages: 74679564 kB
Mapped: 4465992 kB
Shmem: 176 kB
Slab: 65894636 kB
SReclaimable: 40123808 kB
SUnreclaim: 25770828 kB
KernelStack: 23616 kB
PageTables: 216640 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 99170056 kB
Committed_AS: 75530612 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 715408 kB
VmallocChunk: 34359015920 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 10508 kB
DirectMap2M: 1939456 kB
DirectMap1G: 199229440 kB
#### slabtop
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
193075920 193075920 100% 0.19K 9653796 20 38615184K dentry
193031130 193031130 99% 0.12K 6434371 30 25737484K size-128
12781095 12779631 99% 0.10K 345435 37 1381740K buffer_head
214746 214542 99% 0.55K 30678 7 122712K radix_tree_node
46860 46802 99% 0.98K 11715 4 46860K ext4_inode_cache
#### It will create dentry every second.
$ sudo strace -fp 29211 -e trace=open,stat,close,unlink
Process 29211 attached with 370 threads
[pid 33862] stat("/data/druid/var/tmp/druid-groupBy-2c2e19ef-f883-496f-b88d-10dab759eab0_e0677686-1751-43a5-b244-817accd836f8", 0x7efaa8549a10) = -1 ENOENT (No such file or directory)
[pid 33856] stat("/data/druid/var/tmp/druid-groupBy-4033e0b6-2878-4fbc-b941-ab11f075e468_8889a78d-7afb-4290-af6c-5047fd499330", 0x7efaa8b4fb10) = -1 ENOENT (No such file or directory)
[pid 33881] stat("/data/druid/var/tmp/druid-groupBy-e944fe64-64b6-42db-96b3-27857fef7dc1_0490c768-3b39-4b37-b413-d84a03de7025", 0x7efa6b1efb80) = -1 ENOENT (No such file or directory)
[pid 33936] stat("/data/druid/var/tmp/druid-groupBy-b7452444-8665-482b-bec6-42de3abd756e_800e1931-f294-44b1-b844-e4bbec77aff5", 0x7ef9679f7af0) = -1 ENOENT (No such file or directory)
[pid 33890] stat("/data/druid/var/tmp/druid-groupBy-a72b752a-d047-4438-8d3a-8d4f30b98699_7d6382f6-62eb-4e47-85e7-7060014742c2", 0x7efa6a8e6800) = -1 ENOENT (No such file or directory)
[pid 33885] stat("/data/druid/var/tmp/druid-groupBy-96f1429b-53e5-4893-a084-19890e2c1ce7_c0a62ded-af6f-4ed2-a486-2489f639795f", 0x7efa6adeb970) = -1 ENOENT (No such file or directory)
[pid 33869] stat("/data/druid/var/tmp/druid-groupBy-ad823d55-a654-424c-9f96-6c7cd3fb8cef_03746689-7ea4-4d0a-aaf8-a99d3e17288d", 0x7efa6bdfb980) = -1 ENOENT (No such file or directory)
[pid 33878] stat("/data/druid/var/tmp/druid-groupBy-867609dd-82a8-43c3-863d-2fe8672f7937_97d3ebb4-cb0a-4e16-ab51-4a830094334e", 0x7efa6b4f29f0) = -1 ENOENT (No such file or directory)
[pid 33880] stat("/data/druid/var/tmp/druid-groupBy-ef4d32c8-a5b9-4557-b712-bfc7f1589cf7_9e82521f-32d5-496a-9c2d-2a9e187e67dd", 0x7efa6b2f0b10) = -1 ENOENT (No such file or directory)
[pid 33875] stat("/data/druid/var/tmp/druid-groupBy-f6398511-ed75-4547-8d02-c7004f38c1f1_6078cf5e-ef53-4c60-8f57-6413937a0fed", 0x7efa6b7f5880) = -1 ENOENT (No such file or directory)
[pid 33927] stat("/data/druid/var/tmp/druid-groupBy-f3647873-65bc-4d5f-8e3b-ae74781b4601_9154c59f-456a-4d69-a5d9-9ef90d60d1b5", 0x7efa683c1a80) = -1 ENOENT (No such file or directory)
[pid 33942] stat("/data/druid/var/tmp/druid-groupBy-c46029f8-1a26-4782-9413-6af301907289_14b7d398-b600-45c3-9861-46306f20e7ee", 0x7ef9673f1a10) = -1 ENOENT (No such file or directory)
[pid 33861] stat("/data/druid/var/tmp/druid-groupBy-ce632ef2-25a6-4ddb-b09e-67acd1708497_7d188a95-1fb6-4a50-99a3-73ee49d08f53", 0x7efaa864a990) = -1 ENOENT (No such file or directory)
[pid 33897] stat("/data/druid/var/tmp/druid-groupBy-963f0eb8-cdae-4fd4-a119-1152db5effd3_2f33c650-7f91-4e7f-a7b2-ba992a328461", 0x7efa6a1dfb90) = -1 ENOENT (No such file or directory)
[pid 33867] stat("/data/druid/var/tmp/druid-groupBy-d410a69b-a283-475d-a977-f4a5e04ad47a_431f471e-e02a-4fc1-a7ed-ab658215575a", 0x7efa6bffd880) = -1 ENOENT (No such file or directory)
[pid 33903] stat("/data/druid/var/tmp/druid-groupBy-821bd461-493a-4b8d-a948-96733175c553_fdadd570-0dd5-4e97-b7e9-cefc8e06cc15", 0x7efa69bd9a90) = -1 ENOENT (No such file or directory)
[pid 33899] stat("/data/druid/var/tmp/druid-groupBy-e4133c0c-e2ef-46ad-9533-ead801d83491_9e1ababa-4baf-4858-a332-a080bd70a97d", 0x7efa69fdd880) = -1 ENOENT (No such file or directory)
### Root Cause
In the LimitedTemporaryStorage::close() method, if will check directory existence whatever the directory if created
@Override
public void close()
{
synchronized (files) {
if (closed) {
return;
}
closed = true;
for (File file : ImmutableSet.copyOf(files)) {
delete(file);
}
files.clear();
if (storageDirectory.exists() && !storageDirectory.delete()) {
log.warn("Cannot delete storageDirectory: %s", storageDirectory);
}
}
}
### Solution
I will add a flag to check if the directory created.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org