You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "taoyunxing (JIRA)" <ji...@apache.org> on 2016/05/06 05:35:12 UTC
[jira] [Updated] (TS-4279) ats fallen into dead loop for cache
directory overflow
[ https://issues.apache.org/jira/browse/TS-4279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
taoyunxing updated TS-4279:
---------------------------
Description:
CPU: 40 cores, Mem: 120GB, Disk: 1*300GB sys + 11 * 899GB data(naked), OS: CentOS 6.6, ATS: 5.3.1
records.config:
CONFIG proxy.config.cache.min_average_object_size INT 1048576
CONFIG proxy.config.cache.ram_cache.algorithm INT 1
CONFIG proxy.config.cache.ram_cache_cutoff INT 4194304
CONFIG proxy.config.cache.ram_cache.size INT 64424509440
storage.config:
/dev/sdc id=cache.disk.1
I encountered a kind of dead loop situation of ats 5.3.1 on two production hosts, a burst of warning is seen by me in the diags.log for a long time like this:
{code}
[Mar 16 13:04:32.730] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.732] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.733] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.735] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.737] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.739] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.742] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.744] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.747] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.750] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.753] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.756] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
{code}
ats restart in every serval hours, and the TIMEWAIT count is huge above the ESTABLISH TCP connection count.
the following is the current dir snapshot of disk /dev/sdc on one of the hosts:
{code}
Directory for [cache.disk.1 172032:109741163]
Bytes: 8573600
Segments: 14
Buckets: 15310
Entries: 857360
Full: 852904
Empty: 4085
Stale: 0
Free: 371
Bucket Fullness: 4085 15800 32044 41621
42175 33137 22232 12605
Segment Fullness: 60903 60918 60914 60947 60956
60947 60872 60943 60918 60927
60858 60917 60927 60957
Freelist Fullness: 45 30 27 13 0
7 89 5 32 12
83 0 20 8
{code}
I wonder why the value of freelist[4] is zero, which cause ats dead loop, anyone help me? thanks a lot.
was:
CPU: 40 cores, Mem: 120GB, Disk: 1*300GB sys + 11 * 899GB data(naked), OS: CentOS 6.6, ATS: 5.3.1
records.config:
CONFIG proxy.config.cache.min_average_object_size INT 1048576
CONFIG proxy.config.cache.ram_cache.algorithm INT 1
CONFIG proxy.config.cache.ram_cache_cutoff INT 4194304
CONFIG proxy.config.cache.ram_cache.size INT 64424509440
storage.config:
/dev/sdc id=cache.disk.1
I encountered a kind of dead loop situation of ats 5.3.1 on two production hosts, a burst of warning is seen by me in the diags.log for a long time like this:
{code}
[Mar 16 13:04:32.730] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.732] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.733] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.735] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.737] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.739] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.742] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.744] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.747] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.750] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.753] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
[Mar 16 13:04:32.756] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
{code}
ats restart in every serval hours, and the TIMEWAIT count is huge above the ESTABLISH TCP connection count.
the following is the current dir snapshot of disk /dev/sdc on one of the hosts:
{code}
Directory for [cache.disk.1 172032:109741163]
Bytes: 8573600
Segments: 14
Buckets: 15310
Entries: 857360
Full: 852904
Empty: 4085
Stale: 0
Free: 371
Bucket Fullness: 4085 15800 32044 41621
42175 33137 22232 12605
Segment Fullness: 60903 60918 60914 60947 60956
60947 60872 60943 60918 60927
60858 60917 60927 60957
Freelist Fullness: 45 30 27 13 0
7 89 5 32 12
83 0 20 8
{code}
I wonder why the value of freelist[4] is zero, which cause ats dead loop, anyone help me? thinks a lot.
> ats fallen into dead loop for cache directory overflow
> ------------------------------------------------------
>
> Key: TS-4279
> URL: https://issues.apache.org/jira/browse/TS-4279
> Project: Traffic Server
> Issue Type: Bug
> Components: Cache
> Affects Versions: 5.3.1
> Reporter: taoyunxing
>
> CPU: 40 cores, Mem: 120GB, Disk: 1*300GB sys + 11 * 899GB data(naked), OS: CentOS 6.6, ATS: 5.3.1
> records.config:
> CONFIG proxy.config.cache.min_average_object_size INT 1048576
> CONFIG proxy.config.cache.ram_cache.algorithm INT 1
> CONFIG proxy.config.cache.ram_cache_cutoff INT 4194304
> CONFIG proxy.config.cache.ram_cache.size INT 64424509440
> storage.config:
> /dev/sdc id=cache.disk.1
> I encountered a kind of dead loop situation of ats 5.3.1 on two production hosts, a burst of warning is seen by me in the diags.log for a long time like this:
> {code}
> [Mar 16 13:04:32.730] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
> [Mar 16 13:04:32.732] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
> [Mar 16 13:04:32.733] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
> [Mar 16 13:04:32.735] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
> [Mar 16 13:04:32.737] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
> [Mar 16 13:04:32.739] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
> [Mar 16 13:04:32.742] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
> [Mar 16 13:04:32.744] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
> [Mar 16 13:04:32.747] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
> [Mar 16 13:04:32.750] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
> [Mar 16 13:04:32.753] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
> [Mar 16 13:04:32.756] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502 (freelist_clean)> cache directory overflow on '/dev/sdc' segment 4, purging...
> {code}
> ats restart in every serval hours, and the TIMEWAIT count is huge above the ESTABLISH TCP connection count.
> the following is the current dir snapshot of disk /dev/sdc on one of the hosts:
> {code}
> Directory for [cache.disk.1 172032:109741163]
> Bytes: 8573600
> Segments: 14
> Buckets: 15310
> Entries: 857360
> Full: 852904
> Empty: 4085
> Stale: 0
> Free: 371
> Bucket Fullness: 4085 15800 32044 41621
> 42175 33137 22232 12605
>
> Segment Fullness: 60903 60918 60914 60947 60956
> 60947 60872 60943 60918 60927
> 60858 60917 60927 60957
> Freelist Fullness: 45 30 27 13 0
> 7 89 5 32 12
> 83 0 20 8
> {code}
> I wonder why the value of freelist[4] is zero, which cause ats dead loop, anyone help me? thanks a lot.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)