You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by "Ye, Hong" <Ho...@cable.comcast.com> on 2015/11/24 17:12:58 UTC
ATS does not log traffic any more -- WARNING: Access logging to local
log directory suspended - configured space allocation exhausted.
Hi dev team,
I need some help to figure out why one of my server stopped logging traffic. I saw that warning in digs.log. But I have plenty of disk space on the server.
I tried to bounce the service and reboot server. But still, I do not see the log file grown gin size.
Hope someone can help to take a look and let me know how I can fix it. I am using an older version of ATS: 4.0.1.
Thank you very much!
Following is from digs.log:
[Nov 24 15:19:53.240] {0x2b226a90b560} STATUS: opened /var/log/trafficserver/diags.log
[Nov 24 15:19:53.240] {0x2b226a90b560} NOTE: updated diags config
[Nov 24 15:19:53.258] Server {0x2b226a90b560} NOTE: cache clustering disabled
[Nov 24 15:19:53.282] Server {0x2b226a90b560} NOTE: ip_allow.config updated, reloading
[Nov 24 15:19:53.316] Server {0x2b226a90b560} NOTE: cache clustering disabled
[Nov 24 15:19:53.316] Server {0x2b226a90b560} NOTE: logging initialized[15], logging_mode = 3
[Nov 24 15:19:53.317] Server {0x2b226a90b560} WARNING: Access logging to local log directory suspended - configured space allocation exhausted.
[Nov 24 15:19:53.320] Server {0x2b226a90b560} NOTE: loading plugin '/libexec/trafficserver/stats_over_http.so'
[Nov 24 15:19:53.410] Server {0x2b2271d1c700} NOTE: cache enabled
[Nov 24 15:19:53.480] Server {0x2b226a90b560} NOTE: traffic server running
[Nov 24 15:53:47.634] {0x2ae7753a4560} STATUS: opened /var/log/trafficserver/diags.log
[Nov 24 15:53:47.641] {0x2ae7753a4560} NOTE: updated diags config
[Nov 24 15:53:47.658] Server {0x2ae7753a4560} NOTE: cache clustering disabled
[Nov 24 15:53:47.703] Server {0x2ae7753a4560} NOTE: ip_allow.config updated, reloading
[Nov 24 15:53:47.740] Server {0x2ae7753a4560} NOTE: cache clustering disabled
[Nov 24 15:53:47.742] Server {0x2ae7753a4560} NOTE: logging initialized[15], logging_mode = 3
[Nov 24 15:53:47.760] Server {0x2ae7753a4560} WARNING: Access logging to local log directory suspended - configured space allocation exhausted.
[Nov 24 15:53:47.782] Server {0x2ae7753a4560} NOTE: loading plugin '/libexec/trafficserver/stats_over_http.so'
[Nov 24 15:53:47.886] Server {0x2ae77c807700} NOTE: cache enabled
[Nov 24 15:53:48.059] Server {0x2ae7753a4560} NOTE: traffic server running
==== ATS version/build
Apache Traffic Server - traffic_line - 4.0.1 - (build # 0816 on Jan 8 2014 at 16:59:34)
======= logs_xml.config
<LogFormat>
<Name = "cc"/>
<Format = "%<cqtd> %<cqtt> %<chi> %<pssc> %<cfsc> %<cqus>://%<{Host}cqh> %<{Location}psh>"/>
</LogFormat>
<LogObject>
<Format = "cc"/>
<Filename = "cclog"/>
<Mode = "ascii"/>
<Protocols = "http"/>
<RollingEnabled = "1"/>
<RollingIntervalSec ="43200"/>
</LogObject>
<LogFormat>
<Name = "welf"/>
<Format = "id=firewall time=\"%<cqtd> %<cqtt>\" fw=%<phn> pri=6 proto=%<cqus> duration=%<ttmsf> sent=%<psql> rcvd=%<cqhl> src=%<chi> dst=%<shi> dstname=%<shn> user=%<caun> op=%<cqhm> arg=\"%<cqup>\" result=%<pssc> ref=\"%<{Referer}cqh>\" agent=\"%<{user-agent}cqh>\" cache=%<crc>"/>
</LogFormat>
<LogFormat>
<Name = "squid_seconds_only_timestamp"/>
<Format = "%<cqts> %<ttms> %<chi> %<crc>/%<pssc> %<psql> %<cqhm> %<cquc> %<caun> %<phr>/%<pqsn> %<psct>"/>
</LogFormat>
===== disk space status
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/RootVolGroup00-lv_root
7.9G 2.2G 5.4G 29% /
tmpfs 36G 0 36G 0% /dev/shm
/dev/sda1 504M 59M 420M 13% /boot
/dev/mapper/RootVolGroup00-lv_opt
7.9G 166M 7.4G 3% /opt
/dev/mapper/RootVolGroup00-lv_var
400G 27G 352G 8% /var
Kind Regards
Hong Ye
DNS Dev & Ops, Technology and Platform
Desk: 215-286-4488 Cell: 267-273-8368
Re: ATS does not log traffic any more -- WARNING: Access logging to
local log directory suspended - configured space allocation exhausted.
Posted by "Ye, Hong" <Ho...@cable.comcast.com>.
I have following in records.config: (pretty much default values)
CONFIG proxy.config.log.logging_enabled INT 3
CONFIG proxy.config.log.max_secs_per_buffer INT 1
CONFIG proxy.config.log.max_space_mb_for_logs INT 25000
CONFIG proxy.config.log.max_space_mb_for_orphan_logs INT 25
CONFIG proxy.config.log.max_space_mb_headroom INT 1000
CONFIG proxy.config.log.hostname STRING localhost
Is "logspace diagnostic² tag new? I am not seeing it in the config file. I
am running 4.0.1.
Kind Regards
Hong Ye
DNS Dev & Ops, Technology and Platform
Desk: 215-286-4488 Cell: 267-273-8368
On 11/24/15, 1:23 PM, "James Peach" <jp...@apache.org> wrote:
>
>> On Nov 24, 2015, at 8:12 AM, Ye, Hong <Ho...@cable.comcast.com> wrote:
>>
>> Hi dev team,
>>
>> I need some help to figure out why one of my server stopped logging
>>traffic. I saw that warning in digs.log. But I have plenty of disk space
>>on the server.
>> I tried to bounce the service and reboot server. But still, I do not
>>see the log file grown gin size.
>>
>> Hope someone can help to take a look and let me know how I can fix it.
>>I am using an older version of ATS: 4.0.1.
>
>You need to look at your log space configuration:
>
>~$ traffic_ctl config match proxy.config.log
>proxy.config.log.logging_enabled: 3
>proxy.config.log.max_space_mb_for_logs: 25000
>proxy.config.log.max_space_mb_headroom: 1000
>proxy.config.log.rolling_enabled: 1
>proxy.config.log.rolling_interval_sec: 86400
>proxy.config.log.rolling_size_mb: 10
>proxy.config.log.auto_delete_rolled_files: 1
>...
>
>Specifically make sure that proxy.config.log.max_space_mb_for_logs and
>proxy.config.log.max_space_mb_headroom make sense. If you enable the
>"logspace diagnostic tag you should also get some more information.
>
>>
>> Thank you very much!
>>
>>
>> Following is from digs.log:
>>
>>
>> [Nov 24 15:19:53.240] {0x2b226a90b560} STATUS: opened
>>/var/log/trafficserver/diags.log
>> [Nov 24 15:19:53.240] {0x2b226a90b560} NOTE: updated diags config
>> [Nov 24 15:19:53.258] Server {0x2b226a90b560} NOTE: cache clustering
>>disabled
>> [Nov 24 15:19:53.282] Server {0x2b226a90b560} NOTE: ip_allow.config
>>updated, reloading
>> [Nov 24 15:19:53.316] Server {0x2b226a90b560} NOTE: cache clustering
>>disabled
>> [Nov 24 15:19:53.316] Server {0x2b226a90b560} NOTE: logging
>>initialized[15], logging_mode = 3
>> [Nov 24 15:19:53.317] Server {0x2b226a90b560} WARNING: Access logging
>>to local log directory suspended - configured space allocation exhausted.
>> [Nov 24 15:19:53.320] Server {0x2b226a90b560} NOTE: loading plugin
>>'/libexec/trafficserver/stats_over_http.so'
>> [Nov 24 15:19:53.410] Server {0x2b2271d1c700} NOTE: cache enabled
>> [Nov 24 15:19:53.480] Server {0x2b226a90b560} NOTE: traffic server
>>running
>> [Nov 24 15:53:47.634] {0x2ae7753a4560} STATUS: opened
>>/var/log/trafficserver/diags.log
>> [Nov 24 15:53:47.641] {0x2ae7753a4560} NOTE: updated diags config
>> [Nov 24 15:53:47.658] Server {0x2ae7753a4560} NOTE: cache clustering
>>disabled
>> [Nov 24 15:53:47.703] Server {0x2ae7753a4560} NOTE: ip_allow.config
>>updated, reloading
>> [Nov 24 15:53:47.740] Server {0x2ae7753a4560} NOTE: cache clustering
>>disabled
>> [Nov 24 15:53:47.742] Server {0x2ae7753a4560} NOTE: logging
>>initialized[15], logging_mode = 3
>> [Nov 24 15:53:47.760] Server {0x2ae7753a4560} WARNING: Access logging
>>to local log directory suspended - configured space allocation exhausted.
>> [Nov 24 15:53:47.782] Server {0x2ae7753a4560} NOTE: loading plugin
>>'/libexec/trafficserver/stats_over_http.so'
>> [Nov 24 15:53:47.886] Server {0x2ae77c807700} NOTE: cache enabled
>> [Nov 24 15:53:48.059] Server {0x2ae7753a4560} NOTE: traffic server
>>running
>>
>> ==== ATS version/build
>>
>> Apache Traffic Server - traffic_line - 4.0.1 - (build # 0816 on Jan 8
>>2014 at 16:59:34)
>>
>>
>> ======= logs_xml.config
>>
>> <LogFormat>
>> <Name = "cc"/>
>> <Format = "%<cqtd> %<cqtt> %<chi> %<pssc> %<cfsc>
>>%<cqus>://%<{Host}cqh> %<{Location}psh>"/>
>> </LogFormat>
>>
>> <LogObject>
>> <Format = "cc"/>
>> <Filename = "cclog"/>
>> <Mode = "ascii"/>
>> <Protocols = "http"/>
>> <RollingEnabled = "1"/>
>> <RollingIntervalSec ="43200"/>
>> </LogObject>
>>
>> <LogFormat>
>> <Name = "welf"/>
>> <Format = "id=firewall time=\"%<cqtd> %<cqtt>\" fw=%<phn> pri=6
>>proto=%<cqus> duration=%<ttmsf> sent=%<psql> rcvd=%<cqhl> src=%<chi>
>>dst=%<shi> dstname=%<shn> user=%<caun> op=%<cqhm> arg=\"%<cqup>\"
>>result=%<pssc> ref=\"%<{Referer}cqh>\" agent=\"%<{user-agent}cqh>\"
>>cache=%<crc>"/>
>> </LogFormat>
>>
>> <LogFormat>
>> <Name = "squid_seconds_only_timestamp"/>
>> <Format = "%<cqts> %<ttms> %<chi> %<crc>/%<pssc> %<psql> %<cqhm>
>>%<cquc> %<caun> %<phr>/%<pqsn> %<psct>"/>
>> </LogFormat>
>>
>>
>> ===== disk space status
>> df -h
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/mapper/RootVolGroup00-lv_root
>> 7.9G 2.2G 5.4G 29% /
>> tmpfs 36G 0 36G 0% /dev/shm
>> /dev/sda1 504M 59M 420M 13% /boot
>> /dev/mapper/RootVolGroup00-lv_opt
>> 7.9G 166M 7.4G 3% /opt
>> /dev/mapper/RootVolGroup00-lv_var
>> 400G 27G 352G 8% /var
>>
>>
>>
>>
>> Kind Regards
>>
>> Hong Ye
>> DNS Dev & Ops, Technology and Platform
>> Desk: 215-286-4488 Cell: 267-273-8368
>>
>
>
Re: ATS does not log traffic any more -- WARNING: Access logging to local log directory suspended - configured space allocation exhausted.
Posted by James Peach <jp...@apache.org>.
> On Nov 24, 2015, at 8:12 AM, Ye, Hong <Ho...@cable.comcast.com> wrote:
>
> Hi dev team,
>
> I need some help to figure out why one of my server stopped logging traffic. I saw that warning in digs.log. But I have plenty of disk space on the server.
> I tried to bounce the service and reboot server. But still, I do not see the log file grown gin size.
>
> Hope someone can help to take a look and let me know how I can fix it. I am using an older version of ATS: 4.0.1.
You need to look at your log space configuration:
~$ traffic_ctl config match proxy.config.log
proxy.config.log.logging_enabled: 3
proxy.config.log.max_space_mb_for_logs: 25000
proxy.config.log.max_space_mb_headroom: 1000
proxy.config.log.rolling_enabled: 1
proxy.config.log.rolling_interval_sec: 86400
proxy.config.log.rolling_size_mb: 10
proxy.config.log.auto_delete_rolled_files: 1
...
Specifically make sure that proxy.config.log.max_space_mb_for_logs and proxy.config.log.max_space_mb_headroom make sense. If you enable the "logspace diagnostic tag you should also get some more information.
>
> Thank you very much!
>
>
> Following is from digs.log:
>
>
> [Nov 24 15:19:53.240] {0x2b226a90b560} STATUS: opened /var/log/trafficserver/diags.log
> [Nov 24 15:19:53.240] {0x2b226a90b560} NOTE: updated diags config
> [Nov 24 15:19:53.258] Server {0x2b226a90b560} NOTE: cache clustering disabled
> [Nov 24 15:19:53.282] Server {0x2b226a90b560} NOTE: ip_allow.config updated, reloading
> [Nov 24 15:19:53.316] Server {0x2b226a90b560} NOTE: cache clustering disabled
> [Nov 24 15:19:53.316] Server {0x2b226a90b560} NOTE: logging initialized[15], logging_mode = 3
> [Nov 24 15:19:53.317] Server {0x2b226a90b560} WARNING: Access logging to local log directory suspended - configured space allocation exhausted.
> [Nov 24 15:19:53.320] Server {0x2b226a90b560} NOTE: loading plugin '/libexec/trafficserver/stats_over_http.so'
> [Nov 24 15:19:53.410] Server {0x2b2271d1c700} NOTE: cache enabled
> [Nov 24 15:19:53.480] Server {0x2b226a90b560} NOTE: traffic server running
> [Nov 24 15:53:47.634] {0x2ae7753a4560} STATUS: opened /var/log/trafficserver/diags.log
> [Nov 24 15:53:47.641] {0x2ae7753a4560} NOTE: updated diags config
> [Nov 24 15:53:47.658] Server {0x2ae7753a4560} NOTE: cache clustering disabled
> [Nov 24 15:53:47.703] Server {0x2ae7753a4560} NOTE: ip_allow.config updated, reloading
> [Nov 24 15:53:47.740] Server {0x2ae7753a4560} NOTE: cache clustering disabled
> [Nov 24 15:53:47.742] Server {0x2ae7753a4560} NOTE: logging initialized[15], logging_mode = 3
> [Nov 24 15:53:47.760] Server {0x2ae7753a4560} WARNING: Access logging to local log directory suspended - configured space allocation exhausted.
> [Nov 24 15:53:47.782] Server {0x2ae7753a4560} NOTE: loading plugin '/libexec/trafficserver/stats_over_http.so'
> [Nov 24 15:53:47.886] Server {0x2ae77c807700} NOTE: cache enabled
> [Nov 24 15:53:48.059] Server {0x2ae7753a4560} NOTE: traffic server running
>
> ==== ATS version/build
>
> Apache Traffic Server - traffic_line - 4.0.1 - (build # 0816 on Jan 8 2014 at 16:59:34)
>
>
> ======= logs_xml.config
>
> <LogFormat>
> <Name = "cc"/>
> <Format = "%<cqtd> %<cqtt> %<chi> %<pssc> %<cfsc> %<cqus>://%<{Host}cqh> %<{Location}psh>"/>
> </LogFormat>
>
> <LogObject>
> <Format = "cc"/>
> <Filename = "cclog"/>
> <Mode = "ascii"/>
> <Protocols = "http"/>
> <RollingEnabled = "1"/>
> <RollingIntervalSec ="43200"/>
> </LogObject>
>
> <LogFormat>
> <Name = "welf"/>
> <Format = "id=firewall time=\"%<cqtd> %<cqtt>\" fw=%<phn> pri=6 proto=%<cqus> duration=%<ttmsf> sent=%<psql> rcvd=%<cqhl> src=%<chi> dst=%<shi> dstname=%<shn> user=%<caun> op=%<cqhm> arg=\"%<cqup>\" result=%<pssc> ref=\"%<{Referer}cqh>\" agent=\"%<{user-agent}cqh>\" cache=%<crc>"/>
> </LogFormat>
>
> <LogFormat>
> <Name = "squid_seconds_only_timestamp"/>
> <Format = "%<cqts> %<ttms> %<chi> %<crc>/%<pssc> %<psql> %<cqhm> %<cquc> %<caun> %<phr>/%<pqsn> %<psct>"/>
> </LogFormat>
>
>
> ===== disk space status
> df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/RootVolGroup00-lv_root
> 7.9G 2.2G 5.4G 29% /
> tmpfs 36G 0 36G 0% /dev/shm
> /dev/sda1 504M 59M 420M 13% /boot
> /dev/mapper/RootVolGroup00-lv_opt
> 7.9G 166M 7.4G 3% /opt
> /dev/mapper/RootVolGroup00-lv_var
> 400G 27G 352G 8% /var
>
>
>
>
> Kind Regards
>
> Hong Ye
> DNS Dev & Ops, Technology and Platform
> Desk: 215-286-4488 Cell: 267-273-8368
>