You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by GitBox <gi...@apache.org> on 2020/09/17 19:40:04 UTC
[GitHub] [trafficserver] elsloo opened a new issue #7204: Rolled logs are not removed when max_space_mb reached
elsloo opened a new issue #7204:
URL: https://github.com/apache/trafficserver/issues/7204
Log files previously rolled by ATS are not being removed when disk limits specified by the following parameters are reached:
```
proxy.config.log.max_space_mb_for_logs
proxy.config.log.max_space_mb_headroom
```
..and when `proxy.config.log.auto_delete_rolled_files` is set to 1.
This issue could be related to #6932. The configuration works in ATS 8.x, but does not work in ATS 9. Both `rolling_max_count` and `rolling_min_count` are set to zero. Setting the former to `> 0` does not appear to have an effect, and the latter causes files to be rotated as expected, but prior to disk space exhaustion. It could be that the two settings influence the behavior in an unexpected manner, but our expectation is that the `max_space` parameters would work regardless of the min and max count settings.
Additionally, it appears the `proxy.config.log.max_space_mb_for_orphan_log` setting was removed, and the concern is that logs created by a prior running instance of ATS may not be selected as a candidate for removal. I was unable to confirm if this is the case because ATS is not even deleting the files it rolled during its current run.
In the following example, ATS 9 was put under load such that logging would eventually cross the thresholds specified by the above parameters. The parameters do cause ATS to detect that disk space has been exhausted, as seen in this sequence, but ATS is unable to find the logs it had just rolled itself. The log directory was filled by ATS' usual log rotation, creating a number of rotated files as expected and filling up the disk. That is where the log messages begin.
```
...manually removed all rolled files...
NOTE: Logging space is no longer exhausted.
NOTE: Logging disk is no longer low; access logging to local log directory resumed.
...load sent to ATS...
...ATS log files roll as expected...
NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
NOTE: Logging space exhausted, any logs writing to local disk will be dropped!
WARNING: Access logging to local log directory suspended - configured space allocation almost exhausted.
NOTE: [Alarms::signalAlarm] Skipping Alarm: 'Access logging to local log directory suspended - configured space allocation almost exhausted.'
NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
...continues until files are manually removed...
```
The `LogConfig::update_space_used()` method is where the files are detected for removal. Logic wraps the call to `RolledLogDeleter`'s `consider_for_candidacy()` that could be failing, but `consider_for_candidacy()` also has a side effect where it's incrementing the candidate counter, that is then used later in `update_space_used()` via a call to `RolledLogDeleter`'s `has_candidates()` method. That method returns false and prints the above error message (Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion) We can infer that means that `consider_for_candidacy()` never executed, or it did, but didn't find a candidate, and didn't increment the candidate counter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bneradt commented on issue #7204: Rolled logs are not removed when max_space_mb reached
Posted by GitBox <gi...@apache.org>.
bneradt commented on issue #7204:
URL: https://github.com/apache/trafficserver/issues/7204#issuecomment-697947835
I see what is happening now. When a config reload happens, this is what takes place (among a lot of other stuff, of course, this is just what is relevant to this bug):
1. `Log::change_configuration` creates a new LogConfig object and initializes it from the old one: [Log.cc:117](https://github.com/apache/trafficserver/blob/bd089772b735bc374c0f0f4fedccabd1e6a30cda/proxy/logging/Log.cc#L117)
1. In that process of initializing the new LogConfig, LogObject instances are created from the logging.yaml file. Each of those instances dutifully try to register themselves for auto deletion in `LogObject::_setup_rolling` ([LogObject.cc:692](https://github.com/apache/trafficserver/blob/bd089772b735bc374c0f0f4fedccabd1e6a30cda/proxy/logging/LogObject.cc#L692)) via `Log::config->register_rolled_log_auto_delete(m_basename, m_min_rolled);`
1. That Log::config is a global reference that has not yet been set to the new LogConfig object, however. It will be updated later in Log.cc after the init() call from step 1 completes at [Log.cc:120](https://github.com/apache/trafficserver/blob/bd089772b735bc374c0f0f4fedccabd1e6a30cda/proxy/logging/Log.cc#L120), but that's too late for the LogObject's auto delete registration.
Thus the old Log::config gets registered for auto deletion with the new logging.yaml LogObject instances instead and the new LogConfig. As a side note, traffic.out, manager.out, etc. are registered by LogConfig itself, so that is why they were still properly handled after a reload. But none of the logging.yaml objects persist across a reload.
I believe this can be easily resolved in the LogConfig::init() function by copying over the old RolledLogDeleter to the new config.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bneradt commented on issue #7204: Rolled logs are not removed when max_space_mb reached
Posted by GitBox <gi...@apache.org>.
bneradt commented on issue #7204:
URL: https://github.com/apache/trafficserver/issues/7204#issuecomment-698001250
Digging further, what is described here as a problem with auto deletion using the old config looks like just one of a host of like problems that happen as LogObject and LogBuffer objects are created based upon values from the old config. Basically, all objects created in new_config->init in [Log.cc](https://github.com/apache/trafficserver/blob/bd089772b735bc374c0f0f4fedccabd1e6a30cda/proxy/logging/Log.cc#L117) are created referencing the old rather than the new config (the `// <--` comments are added as annotations by me):
```
new_config = new LogConfig;
ink_assert(new_config != nullptr);
new_config->read_configuration_variables(); // <-- New config has new user-supplied values.
// grab the _APImutex so we can transfer the api objects to
// the new config
//
ink_mutex_acquire(prev->log_object_manager._APImutex);
Debug("log-api-mutex", "Log::change_configuration acquired api mutex");
new_config->init(Log::config); // <-- LogObject and LogBuffer objects are created here,
// but as they are created they reference the old Log::config.
// Make the new LogConfig active.
ink_atomic_swap(&Log::config, new_config); // <-- Only here is Log::config updated.
```
Thus, for example, if the user changes the max buffer size and does a config reload, LogObjects will instead be configured to use the old max buffer size.
While fixing the problem with rolled log auto deletion, we should consider fixing these issues with LogObject and LogBuffer as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bneradt commented on issue #7204: Rolled logs are not removed when max_space_mb reached
Posted by GitBox <gi...@apache.org>.
bneradt commented on issue #7204:
URL: https://github.com/apache/trafficserver/issues/7204#issuecomment-694539466
1. Can you verify you don't see this error message: [Error opening logging directory...](https://github.com/apache/trafficserver/blob/63f8d1d/proxy/logging/LogConfig.cc#L622)
1. Also, would it be possible to enable the "logspace" debug tag? That would provide some helpful diagnostic information. I'd like to verify that you see multiple instances of these: [Registering rotated log deletion...](https://github.com/apache/trafficserver/blob/63f8d1d/proxy/logging/LogConfig.cc#L118)
1. With that information, lets verify that the names of your log files in your log directory should get translated to their unrolled names properly via [LogUtils::get_unrolled_filename](https://github.com/apache/trafficserver/blob/63f8d1d/proxy/logging/LogUtils.cc#L482)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bneradt commented on issue #7204: Rolled logs are not removed when max_space_mb reached
Posted by GitBox <gi...@apache.org>.
bneradt commented on issue #7204:
URL: https://github.com/apache/trafficserver/issues/7204#issuecomment-698619302
PR is up. Thank you @elsloo for your thorough description of the problem. It helped me reproduce the bug.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] elsloo commented on issue #7204: Rolled logs are not removed when max_space_mb reached
Posted by GitBox <gi...@apache.org>.
elsloo commented on issue #7204:
URL: https://github.com/apache/trafficserver/issues/7204#issuecomment-696210488
After further testing, I've narrowed this down to when `logging.yaml` is updated. Once that file is touched, regardless of whether or not a config change is present, and the config is reloaded, log rolling stops working. There is no "Error opening logging directory..." message logged.
To reproduce, I lowered the log rolling thresholds, and directed traffic toward ATS until it started rolling logs. I then updated records.config to enable debug for the `logspace` tag, touched `logging.yaml` and reloaded the config via `traffic_ctl config reload`. I also reproduced the issue by performing the same sequence, but starting with an empty log directory, and reloading the configuration prior to the log directory being deemed full by ATS. The debug output of these steps follow, marked as sequence A and B, respectively.
**Sequence A:**
```
traffic_manager[54905]: traffic_manager NOTE: User has changed config file logging.yaml
traffic_manager[54905]: traffic_manager NOTE: User has changed config file records.config
traffic_manager[54905]: traffic_manager NOTE: updated diags config
traffic_server[54915]: [ET_NET 5] NOTE: updated diags config
traffic_server[54915]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[54905]: [Sep 21 15:44:10.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for diags.log with min roll count 0
traffic_manager[54905]: [Sep 21 15:44:10.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for manager.log with min roll count 0
traffic_manager[54905]: [Sep 21 15:44:10.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for traffic.out with min roll count 0
traffic_manager[54905]: [Sep 21 15:44:10.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 211372870 bytes being used for logs
traffic_manager[54905]: [Sep 21 15:44:10.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5111513088 bytes left on partition
traffic_manager[54905]: [Sep 21 15:44:10.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006655488, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:10.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[54905]: [Sep 21 15:44:10.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006655488, partition headroom 10485760, space is not available
traffic_server[54915]: [LOG_FLUSH] NOTE: Logging space exhausted, any logs writing to local disk will be dropped!
traffic_manager[54905]: [Sep 21 15:44:10.009] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for public_request.log with min roll count 0
traffic_manager[54905]: [Sep 21 15:44:10.009] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for private_request.log with min roll count 0
traffic_server[54915]: [LOG_FLUSH] WARNING: Access logging to local log directory suspended - configured space allocation almost exhausted.
traffic_manager[54905]: [Sep 21 15:44:10.009] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for specific_remap.log with min roll count 0
traffic_manager[54905]: [Sep 21 15:44:10.009] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for alternate.log with min roll count 0
traffic_manager[54905]: [Sep 21 15:44:10.009] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for sequence.log with min roll count 0
traffic_manager[54905]: [Sep 21 15:44:10.009] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for error.log with min roll count 0
traffic_server[54915]: [LOG_FLUSH] NOTE: /path/to/etc/trafficserver/logging.yaml loading ...
traffic_server[54915]: [LOG_FLUSH] NOTE: /path/to/etc/trafficserver/logging.yaml finished loading
traffic_server[57183]: Alarm Message: Access logging to local log directory suspended - configured space allocation almost exhausted.
traffic_server[54915]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[54905]: [Sep 21 15:44:15.741] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006655488, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:15.742] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 211372870 bytes being used for logs
traffic_manager[54905]: [Sep 21 15:44:15.742] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5111508992 bytes left on partition
traffic_manager[54905]: [Sep 21 15:44:15.742] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006651392, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:15.742] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[54905]: [Sep 21 15:44:15.742] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006651392, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:20.736] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006651392, partition headroom 10485760, space is not available
traffic_server[54915]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[54905]: [Sep 21 15:44:20.736] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 211372870 bytes being used for logs
traffic_manager[54905]: [Sep 21 15:44:20.736] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5111508992 bytes left on partition
traffic_manager[54905]: [Sep 21 15:44:20.736] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006651392, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:20.736] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[54905]: [Sep 21 15:44:20.736] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006651392, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:25.731] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006651392, partition headroom 10485760, space is not available
traffic_server[54915]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[54905]: [Sep 21 15:44:25.732] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 211372870 bytes being used for logs
traffic_manager[54905]: [Sep 21 15:44:25.732] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5111504896 bytes left on partition
traffic_manager[54905]: [Sep 21 15:44:25.732] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006647296, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:25.732] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[54905]: [Sep 21 15:44:25.732] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006647296, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:30.747] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006647296, partition headroom 10485760, space is not available
traffic_server[54915]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[54905]: [Sep 21 15:44:30.747] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 211372870 bytes being used for logs
traffic_manager[54905]: [Sep 21 15:44:30.747] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5111504896 bytes left on partition
traffic_manager[54905]: [Sep 21 15:44:30.747] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006647296, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:30.747] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[54905]: [Sep 21 15:44:30.747] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006647296, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:35.765] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006647296, partition headroom 10485760, space is not available
traffic_server[54915]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[54905]: [Sep 21 15:44:35.765] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 211372870 bytes being used for logs
traffic_manager[54905]: [Sep 21 15:44:35.765] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5111500800 bytes left on partition
traffic_manager[54905]: [Sep 21 15:44:35.765] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006643200, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:35.765] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[54905]: [Sep 21 15:44:35.765] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006643200, partition headroom 10485760, space is not available
```
**Sequence B:**
```
traffic_manager[57788]: traffic_manager NOTE: User has changed config file logging.yaml
traffic_manager[57788]: traffic_manager NOTE: User has changed config file records.config
traffic_manager[57788]: traffic_manager NOTE: updated diags config
traffic_server[57797]: [ET_NET 5] NOTE: updated diags config
traffic_manager[57788]: [Sep 21 15:47:24.036] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for diags.log with min roll count 0
traffic_manager[57788]: [Sep 21 15:47:24.036] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for manager.log with min roll count 0
traffic_server[57797]: [LOG_FLUSH] NOTE: /path/to/etc/trafficserver/logging.yaml loading ...
traffic_manager[57788]: [Sep 21 15:47:24.036] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for traffic.out with min roll count 0
traffic_manager[57788]: [Sep 21 15:47:24.036] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 38342439 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:47:24.036] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5284458496 bytes left on partition
traffic_manager[57788]: [Sep 21 15:47:24.036] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 143200039, configured space 314572800, physical space left 5179600896, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:24.036] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 143200039, configured space 314572800, physical space left 5179600896, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:24.037] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for public_request.log with min roll count 0
traffic_manager[57788]: [Sep 21 15:47:24.038] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for private_request.log with min roll count 0
traffic_manager[57788]: [Sep 21 15:47:24.038] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for specific_remap.log with min roll count 0
traffic_server[57797]: [LOG_FLUSH] NOTE: /path/to/etc/trafficserver/logging.yaml finished loading
traffic_manager[57788]: [Sep 21 15:47:24.038] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for alternate.log with min roll count 0
traffic_manager[57788]: [Sep 21 15:47:24.038] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for sequence.log with min roll count 0
traffic_manager[57788]: [Sep 21 15:47:24.038] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for error.log with min roll count 0
traffic_manager[57788]: [Sep 21 15:47:29.003] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 143200039, configured space 314572800, physical space left 5179600896, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 143200039, configured space 314572800, physical space left 5179600896, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 50272588 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:47:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5272522752 bytes left on partition
traffic_manager[57788]: [Sep 21 15:47:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 155130188, configured space 314572800, physical space left 5167665152, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 155130188, configured space 314572800, physical space left 5167665152, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:39.000] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 155130188, configured space 314572800, physical space left 5167665152, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:44.010] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 155130188, configured space 314572800, physical space left 5167665152, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:44.010] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 62371443 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:47:44.011] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5260419072 bytes left on partition
traffic_manager[57788]: [Sep 21 15:47:44.011] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 167229043, configured space 314572800, physical space left 5155561472, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:44.011] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 167229043, configured space 314572800, physical space left 5155561472, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:49.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 167229043, configured space 314572800, physical space left 5155561472, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:54.007] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 167229043, configured space 314572800, physical space left 5155561472, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:54.007] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 74008343 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:47:54.007] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5248778240 bytes left on partition
traffic_manager[57788]: [Sep 21 15:47:54.007] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 178865943, configured space 314572800, physical space left 5143920640, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:54.007] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 178865943, configured space 314572800, physical space left 5143920640, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:59.010] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 178865943, configured space 314572800, physical space left 5143920640, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:04.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 178865943, configured space 314572800, physical space left 5143920640, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:04.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 85811958 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:48:04.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5236969472 bytes left on partition
traffic_manager[57788]: [Sep 21 15:48:04.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 190669558, configured space 314572800, physical space left 5132111872, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:04.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 190669558, configured space 314572800, physical space left 5132111872, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:09.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 190669558, configured space 314572800, physical space left 5132111872, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:14.021] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 190669558, configured space 314572800, physical space left 5132111872, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:14.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 97702052 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:48:14.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5225082880 bytes left on partition
traffic_manager[57788]: [Sep 21 15:48:14.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 202559652, configured space 314572800, physical space left 5120225280, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:14.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 202559652, configured space 314572800, physical space left 5120225280, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:19.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 202559652, configured space 314572800, physical space left 5120225280, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:24.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 202559652, configured space 314572800, physical space left 5120225280, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:24.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 109420096 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:48:24.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5213364224 bytes left on partition
traffic_manager[57788]: [Sep 21 15:48:24.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 214277696, configured space 314572800, physical space left 5108506624, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:24.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 214277696, configured space 314572800, physical space left 5108506624, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:29.089] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 214277696, configured space 314572800, physical space left 5108506624, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:34.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 214277696, configured space 314572800, physical space left 5108506624, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:34.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 121121949 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:48:34.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5201653760 bytes left on partition
traffic_manager[57788]: [Sep 21 15:48:34.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 225979549, configured space 314572800, physical space left 5096796160, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:34.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 225979549, configured space 314572800, physical space left 5096796160, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:39.000] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 225979549, configured space 314572800, physical space left 5096796160, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:44.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 225979549, configured space 314572800, physical space left 5096796160, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:44.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 133484496 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:48:44.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5189292032 bytes left on partition
traffic_manager[57788]: [Sep 21 15:48:44.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 238342096, configured space 314572800, physical space left 5084434432, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:44.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 238342096, configured space 314572800, physical space left 5084434432, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:49.003] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 238342096, configured space 314572800, physical space left 5084434432, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:54.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 238342096, configured space 314572800, physical space left 5084434432, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:54.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 145051899 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:48:54.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5177720832 bytes left on partition
traffic_manager[57788]: [Sep 21 15:48:54.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 249909499, configured space 314572800, physical space left 5072863232, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:54.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 249909499, configured space 314572800, physical space left 5072863232, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:59.000] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 249909499, configured space 314572800, physical space left 5072863232, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:04.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 249909499, configured space 314572800, physical space left 5072863232, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:04.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 156798296 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:49:04.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5165969408 bytes left on partition
traffic_manager[57788]: [Sep 21 15:49:04.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 261655896, configured space 314572800, physical space left 5061111808, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:04.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 261655896, configured space 314572800, physical space left 5061111808, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:09.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 261655896, configured space 314572800, physical space left 5061111808, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:14.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 261655896, configured space 314572800, physical space left 5061111808, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:14.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 168429572 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:49:14.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5154336768 bytes left on partition
traffic_manager[57788]: [Sep 21 15:49:14.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 273287172, configured space 314572800, physical space left 5049479168, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:14.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 273287172, configured space 314572800, physical space left 5049479168, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:19.009] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 273287172, configured space 314572800, physical space left 5049479168, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:24.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 273287172, configured space 314572800, physical space left 5049479168, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:24.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 180251828 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:49:24.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5142507520 bytes left on partition
traffic_manager[57788]: [Sep 21 15:49:24.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 285109428, configured space 314572800, physical space left 5037649920, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:24.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 285109428, configured space 314572800, physical space left 5037649920, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:29.069] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 285109428, configured space 314572800, physical space left 5037649920, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:34.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 285109428, configured space 314572800, physical space left 5037649920, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 192540745 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:49:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5130215424 bytes left on partition
traffic_manager[57788]: [Sep 21 15:49:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 297398345, configured space 314572800, physical space left 5025357824, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 297398345, configured space 314572800, physical space left 5025357824, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:39.019] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 297398345, configured space 314572800, physical space left 5025357824, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:44.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 297398345, configured space 314572800, physical space left 5025357824, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:44.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 204275711 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:49:44.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5118476288 bytes left on partition
traffic_manager[57788]: [Sep 21 15:49:44.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 309133311, configured space 314572800, physical space left 5013618688, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:44.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 309133311, configured space 314572800, physical space left 5013618688, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:49.007] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 309133311, configured space 314572800, physical space left 5013618688, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:54.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 309133311, configured space 314572800, physical space left 5013618688, partition headroom 10485760, space is available
traffic_server[57797]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[57788]: [Sep 21 15:49:54.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 216233067 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:49:54.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5106515968 bytes left on partition
traffic_manager[57788]: [Sep 21 15:49:54.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001658368, partition headroom 10485760, space is not available
traffic_manager[57788]: [Sep 21 15:49:54.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[57788]: [Sep 21 15:49:54.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001658368, partition headroom 10485760, space is not available
traffic_server[57797]: [LOG_FLUSH] NOTE: Logging space exhausted, any logs writing to local disk will be dropped!
traffic_server[57797]: [LOG_FLUSH] WARNING: Access logging to local log directory suspended - configured space allocation almost exhausted.
traffic_server[59578]: Alarm Message: Access logging to local log directory suspended - configured space allocation almost exhausted.
traffic_server[57797]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[57788]: [Sep 21 15:49:59.465] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001658368, partition headroom 10485760, space is not available
traffic_manager[57788]: [Sep 21 15:49:59.465] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 216233067 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:49:59.465] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5106511872 bytes left on partition
traffic_manager[57788]: [Sep 21 15:49:59.465] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001654272, partition headroom 10485760, space is not available
traffic_manager[57788]: [Sep 21 15:49:59.465] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[57788]: [Sep 21 15:49:59.465] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001654272, partition headroom 10485760, space is not available
traffic_manager[57788]: [Sep 21 15:50:04.473] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001654272, partition headroom 10485760, space is not available
traffic_server[57797]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[57788]: [Sep 21 15:50:04.473] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 216233067 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:50:04.473] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5106511872 bytes left on partition
traffic_manager[57788]: [Sep 21 15:50:04.473] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001654272, partition headroom 10485760, space is not available
traffic_manager[57788]: [Sep 21 15:50:04.473] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[57788]: [Sep 21 15:50:04.473] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001654272, partition headroom 10485760, space is not available
traffic_manager[57788]: [Sep 21 15:50:09.477] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001654272, partition headroom 10485760, space is not available
traffic_server[57797]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bneradt closed issue #7204: Rolled logs are not removed when max_space_mb reached
Posted by GitBox <gi...@apache.org>.
bneradt closed issue #7204:
URL: https://github.com/apache/trafficserver/issues/7204
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bneradt commented on issue #7204: Rolled logs are not removed when max_space_mb reached
Posted by GitBox <gi...@apache.org>.
bneradt commented on issue #7204:
URL: https://github.com/apache/trafficserver/issues/7204#issuecomment-697101517
I've reproduced this in our prod sim environment. After touching `logging.yaml` and performing the `traffic_ctl config reload` as described by Jeff above, Traffic Server seems to lose certain categories of rolled log files it can autodelete. It still can delete rolled traffic.out, diags.out, and manager.out log files, for instance, but not any log files configured via logging.yaml. Once I restart the Traffic Server process, it notices the logging.yaml rolled log files and deletes them, freeing up the space it couldn't before after the config reload.
It's not clear to me why this is happening. I see in the traffic.out output that those logging.yaml configured log files are indeed registered for log deletion, so why aren't they being deleted? I'll look into this more tomorrow morning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bneradt commented on issue #7204: Rolled logs are not removed when max_space_mb reached
Posted by GitBox <gi...@apache.org>.
bneradt commented on issue #7204:
URL: https://github.com/apache/trafficserver/issues/7204#issuecomment-700058271
On a related note, rather than using the ATS implemented log rotation and deletion mechanism, Yahoo has been internally using logrotate with the signal USR2 to handle log rotation and deletion for ATS 9. Support for this has been released to open source via:
https://github.com/apache/trafficserver/pull/6806
I mention this since it is an alternative mechanism for handling log rotation that others may want to consider.
It is merged into master and is targeted for 9.1.x.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] elsloo commented on issue #7204: Rolled logs are not removed when max_space_mb reached
Posted by GitBox <gi...@apache.org>.
elsloo commented on issue #7204:
URL: https://github.com/apache/trafficserver/issues/7204#issuecomment-696210488
After further testing, I've narrowed this down to when `logging.yaml` is updated. Once that file is touched, regardless of whether or not a config change is present, and the config is reloaded, log rolling stops working. There is no "Error opening logging directory..." message logged.
To reproduce, I lowered the log rolling thresholds, and directed traffic toward ATS until it started rolling logs. I then updated records.config to enable debug for the `logspace` tag, touched `logging.yaml` and reloaded the config via `traffic_ctl config reload`. I also reproduced the issue by performing the same sequence, but starting with an empty log directory, and reloading the configuration prior to the log directory being deemed full by ATS. The debug output of these steps follow, marked as sequence A and B, respectively.
**Sequence A:**
```
traffic_manager[54905]: traffic_manager NOTE: User has changed config file logging.yaml
traffic_manager[54905]: traffic_manager NOTE: User has changed config file records.config
traffic_manager[54905]: traffic_manager NOTE: updated diags config
traffic_server[54915]: [ET_NET 5] NOTE: updated diags config
traffic_server[54915]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[54905]: [Sep 21 15:44:10.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for diags.log with min roll count 0
traffic_manager[54905]: [Sep 21 15:44:10.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for manager.log with min roll count 0
traffic_manager[54905]: [Sep 21 15:44:10.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for traffic.out with min roll count 0
traffic_manager[54905]: [Sep 21 15:44:10.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 211372870 bytes being used for logs
traffic_manager[54905]: [Sep 21 15:44:10.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5111513088 bytes left on partition
traffic_manager[54905]: [Sep 21 15:44:10.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006655488, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:10.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[54905]: [Sep 21 15:44:10.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006655488, partition headroom 10485760, space is not available
traffic_server[54915]: [LOG_FLUSH] NOTE: Logging space exhausted, any logs writing to local disk will be dropped!
traffic_manager[54905]: [Sep 21 15:44:10.009] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for public_request.log with min roll count 0
traffic_manager[54905]: [Sep 21 15:44:10.009] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for private_request.log with min roll count 0
traffic_server[54915]: [LOG_FLUSH] WARNING: Access logging to local log directory suspended - configured space allocation almost exhausted.
traffic_manager[54905]: [Sep 21 15:44:10.009] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for specific_remap.log with min roll count 0
traffic_manager[54905]: [Sep 21 15:44:10.009] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for alternate.log with min roll count 0
traffic_manager[54905]: [Sep 21 15:44:10.009] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for sequence.log with min roll count 0
traffic_manager[54905]: [Sep 21 15:44:10.009] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for error.log with min roll count 0
traffic_server[54915]: [LOG_FLUSH] NOTE: /path/to/etc/trafficserver/logging.yaml loading ...
traffic_server[54915]: [LOG_FLUSH] NOTE: /path/to/etc/trafficserver/logging.yaml finished loading
traffic_server[57183]: Alarm Message: Access logging to local log directory suspended - configured space allocation almost exhausted.
traffic_server[54915]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[54905]: [Sep 21 15:44:15.741] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006655488, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:15.742] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 211372870 bytes being used for logs
traffic_manager[54905]: [Sep 21 15:44:15.742] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5111508992 bytes left on partition
traffic_manager[54905]: [Sep 21 15:44:15.742] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006651392, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:15.742] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[54905]: [Sep 21 15:44:15.742] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006651392, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:20.736] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006651392, partition headroom 10485760, space is not available
traffic_server[54915]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[54905]: [Sep 21 15:44:20.736] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 211372870 bytes being used for logs
traffic_manager[54905]: [Sep 21 15:44:20.736] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5111508992 bytes left on partition
traffic_manager[54905]: [Sep 21 15:44:20.736] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006651392, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:20.736] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[54905]: [Sep 21 15:44:20.736] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006651392, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:25.731] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006651392, partition headroom 10485760, space is not available
traffic_server[54915]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[54905]: [Sep 21 15:44:25.732] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 211372870 bytes being used for logs
traffic_manager[54905]: [Sep 21 15:44:25.732] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5111504896 bytes left on partition
traffic_manager[54905]: [Sep 21 15:44:25.732] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006647296, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:25.732] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[54905]: [Sep 21 15:44:25.732] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006647296, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:30.747] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006647296, partition headroom 10485760, space is not available
traffic_server[54915]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[54905]: [Sep 21 15:44:30.747] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 211372870 bytes being used for logs
traffic_manager[54905]: [Sep 21 15:44:30.747] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5111504896 bytes left on partition
traffic_manager[54905]: [Sep 21 15:44:30.747] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006647296, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:30.747] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[54905]: [Sep 21 15:44:30.747] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006647296, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:35.765] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006647296, partition headroom 10485760, space is not available
traffic_server[54915]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[54905]: [Sep 21 15:44:35.765] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 211372870 bytes being used for logs
traffic_manager[54905]: [Sep 21 15:44:35.765] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5111500800 bytes left on partition
traffic_manager[54905]: [Sep 21 15:44:35.765] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006643200, partition headroom 10485760, space is not available
traffic_manager[54905]: [Sep 21 15:44:35.765] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[54905]: [Sep 21 15:44:35.765] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 316230470, configured space 314572800, physical space left 5006643200, partition headroom 10485760, space is not available
```
**Sequence B:**
```
traffic_manager[57788]: traffic_manager NOTE: User has changed config file logging.yaml
traffic_manager[57788]: traffic_manager NOTE: User has changed config file records.config
traffic_manager[57788]: traffic_manager NOTE: updated diags config
traffic_server[57797]: [ET_NET 5] NOTE: updated diags config
traffic_manager[57788]: [Sep 21 15:47:24.036] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for diags.log with min roll count 0
traffic_manager[57788]: [Sep 21 15:47:24.036] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for manager.log with min roll count 0
traffic_server[57797]: [LOG_FLUSH] NOTE: /path/to/etc/trafficserver/logging.yaml loading ...
traffic_manager[57788]: [Sep 21 15:47:24.036] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for traffic.out with min roll count 0
traffic_manager[57788]: [Sep 21 15:47:24.036] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 38342439 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:47:24.036] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5284458496 bytes left on partition
traffic_manager[57788]: [Sep 21 15:47:24.036] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 143200039, configured space 314572800, physical space left 5179600896, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:24.036] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 143200039, configured space 314572800, physical space left 5179600896, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:24.037] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for public_request.log with min roll count 0
traffic_manager[57788]: [Sep 21 15:47:24.038] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for private_request.log with min roll count 0
traffic_manager[57788]: [Sep 21 15:47:24.038] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for specific_remap.log with min roll count 0
traffic_server[57797]: [LOG_FLUSH] NOTE: /path/to/etc/trafficserver/logging.yaml finished loading
traffic_manager[57788]: [Sep 21 15:47:24.038] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for alternate.log with min roll count 0
traffic_manager[57788]: [Sep 21 15:47:24.038] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for sequence.log with min roll count 0
traffic_manager[57788]: [Sep 21 15:47:24.038] [LOG_FLUSH] DEBUG: <LogConfig.cc:119 (register_rolled_log_auto_delete)> (logspace) Registering rotated log deletion for error.log with min roll count 0
traffic_manager[57788]: [Sep 21 15:47:29.003] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 143200039, configured space 314572800, physical space left 5179600896, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 143200039, configured space 314572800, physical space left 5179600896, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 50272588 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:47:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5272522752 bytes left on partition
traffic_manager[57788]: [Sep 21 15:47:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 155130188, configured space 314572800, physical space left 5167665152, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 155130188, configured space 314572800, physical space left 5167665152, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:39.000] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 155130188, configured space 314572800, physical space left 5167665152, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:44.010] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 155130188, configured space 314572800, physical space left 5167665152, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:44.010] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 62371443 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:47:44.011] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5260419072 bytes left on partition
traffic_manager[57788]: [Sep 21 15:47:44.011] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 167229043, configured space 314572800, physical space left 5155561472, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:44.011] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 167229043, configured space 314572800, physical space left 5155561472, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:49.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 167229043, configured space 314572800, physical space left 5155561472, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:54.007] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 167229043, configured space 314572800, physical space left 5155561472, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:54.007] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 74008343 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:47:54.007] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5248778240 bytes left on partition
traffic_manager[57788]: [Sep 21 15:47:54.007] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 178865943, configured space 314572800, physical space left 5143920640, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:54.007] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 178865943, configured space 314572800, physical space left 5143920640, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:47:59.010] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 178865943, configured space 314572800, physical space left 5143920640, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:04.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 178865943, configured space 314572800, physical space left 5143920640, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:04.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 85811958 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:48:04.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5236969472 bytes left on partition
traffic_manager[57788]: [Sep 21 15:48:04.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 190669558, configured space 314572800, physical space left 5132111872, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:04.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 190669558, configured space 314572800, physical space left 5132111872, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:09.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 190669558, configured space 314572800, physical space left 5132111872, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:14.021] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 190669558, configured space 314572800, physical space left 5132111872, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:14.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 97702052 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:48:14.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5225082880 bytes left on partition
traffic_manager[57788]: [Sep 21 15:48:14.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 202559652, configured space 314572800, physical space left 5120225280, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:14.022] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 202559652, configured space 314572800, physical space left 5120225280, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:19.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 202559652, configured space 314572800, physical space left 5120225280, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:24.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 202559652, configured space 314572800, physical space left 5120225280, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:24.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 109420096 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:48:24.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5213364224 bytes left on partition
traffic_manager[57788]: [Sep 21 15:48:24.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 214277696, configured space 314572800, physical space left 5108506624, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:24.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 214277696, configured space 314572800, physical space left 5108506624, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:29.089] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 214277696, configured space 314572800, physical space left 5108506624, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:34.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 214277696, configured space 314572800, physical space left 5108506624, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:34.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 121121949 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:48:34.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5201653760 bytes left on partition
traffic_manager[57788]: [Sep 21 15:48:34.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 225979549, configured space 314572800, physical space left 5096796160, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:34.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 225979549, configured space 314572800, physical space left 5096796160, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:39.000] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 225979549, configured space 314572800, physical space left 5096796160, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:44.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 225979549, configured space 314572800, physical space left 5096796160, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:44.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 133484496 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:48:44.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5189292032 bytes left on partition
traffic_manager[57788]: [Sep 21 15:48:44.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 238342096, configured space 314572800, physical space left 5084434432, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:44.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 238342096, configured space 314572800, physical space left 5084434432, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:49.003] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 238342096, configured space 314572800, physical space left 5084434432, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:54.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 238342096, configured space 314572800, physical space left 5084434432, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:54.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 145051899 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:48:54.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5177720832 bytes left on partition
traffic_manager[57788]: [Sep 21 15:48:54.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 249909499, configured space 314572800, physical space left 5072863232, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:54.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 249909499, configured space 314572800, physical space left 5072863232, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:48:59.000] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 249909499, configured space 314572800, physical space left 5072863232, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:04.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 249909499, configured space 314572800, physical space left 5072863232, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:04.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 156798296 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:49:04.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5165969408 bytes left on partition
traffic_manager[57788]: [Sep 21 15:49:04.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 261655896, configured space 314572800, physical space left 5061111808, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:04.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 261655896, configured space 314572800, physical space left 5061111808, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:09.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 261655896, configured space 314572800, physical space left 5061111808, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:14.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 261655896, configured space 314572800, physical space left 5061111808, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:14.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 168429572 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:49:14.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5154336768 bytes left on partition
traffic_manager[57788]: [Sep 21 15:49:14.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 273287172, configured space 314572800, physical space left 5049479168, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:14.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 273287172, configured space 314572800, physical space left 5049479168, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:19.009] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 273287172, configured space 314572800, physical space left 5049479168, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:24.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 273287172, configured space 314572800, physical space left 5049479168, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:24.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 180251828 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:49:24.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5142507520 bytes left on partition
traffic_manager[57788]: [Sep 21 15:49:24.001] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 285109428, configured space 314572800, physical space left 5037649920, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:24.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 285109428, configured space 314572800, physical space left 5037649920, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:29.069] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 285109428, configured space 314572800, physical space left 5037649920, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:34.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 285109428, configured space 314572800, physical space left 5037649920, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 192540745 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:49:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5130215424 bytes left on partition
traffic_manager[57788]: [Sep 21 15:49:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 297398345, configured space 314572800, physical space left 5025357824, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:34.006] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 297398345, configured space 314572800, physical space left 5025357824, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:39.019] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 297398345, configured space 314572800, physical space left 5025357824, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:44.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 297398345, configured space 314572800, physical space left 5025357824, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:44.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 204275711 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:49:44.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5118476288 bytes left on partition
traffic_manager[57788]: [Sep 21 15:49:44.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 309133311, configured space 314572800, physical space left 5013618688, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:44.002] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 309133311, configured space 314572800, physical space left 5013618688, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:49.007] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 309133311, configured space 314572800, physical space left 5013618688, partition headroom 10485760, space is available
traffic_manager[57788]: [Sep 21 15:49:54.004] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 309133311, configured space 314572800, physical space left 5013618688, partition headroom 10485760, space is available
traffic_server[57797]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[57788]: [Sep 21 15:49:54.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 216233067 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:49:54.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5106515968 bytes left on partition
traffic_manager[57788]: [Sep 21 15:49:54.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001658368, partition headroom 10485760, space is not available
traffic_manager[57788]: [Sep 21 15:49:54.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[57788]: [Sep 21 15:49:54.005] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001658368, partition headroom 10485760, space is not available
traffic_server[57797]: [LOG_FLUSH] NOTE: Logging space exhausted, any logs writing to local disk will be dropped!
traffic_server[57797]: [LOG_FLUSH] WARNING: Access logging to local log directory suspended - configured space allocation almost exhausted.
traffic_server[59578]: Alarm Message: Access logging to local log directory suspended - configured space allocation almost exhausted.
traffic_server[57797]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[57788]: [Sep 21 15:49:59.465] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001658368, partition headroom 10485760, space is not available
traffic_manager[57788]: [Sep 21 15:49:59.465] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 216233067 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:49:59.465] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5106511872 bytes left on partition
traffic_manager[57788]: [Sep 21 15:49:59.465] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001654272, partition headroom 10485760, space is not available
traffic_manager[57788]: [Sep 21 15:49:59.465] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[57788]: [Sep 21 15:49:59.465] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001654272, partition headroom 10485760, space is not available
traffic_manager[57788]: [Sep 21 15:50:04.473] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001654272, partition headroom 10485760, space is not available
traffic_server[57797]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
traffic_manager[57788]: [Sep 21 15:50:04.473] [LOG_FLUSH] DEBUG: <LogConfig.cc:665 (update_space_used)> (logspace) 216233067 bytes being used for logs
traffic_manager[57788]: [Sep 21 15:50:04.473] [LOG_FLUSH] DEBUG: <LogConfig.cc:666 (update_space_used)> (logspace) 5106511872 bytes left on partition
traffic_manager[57788]: [Sep 21 15:50:04.473] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001654272, partition headroom 10485760, space is not available
traffic_manager[57788]: [Sep 21 15:50:04.473] [LOG_FLUSH] DEBUG: <LogConfig.cc:683 (update_space_used)> (logspace) headroom reached, trying to clear space ...
traffic_manager[57788]: [Sep 21 15:50:04.473] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001654272, partition headroom 10485760, space is not available
traffic_manager[57788]: [Sep 21 15:50:09.477] [LOG_FLUSH] DEBUG: <LogConfig.cc:561 (space_to_write)> (logspace) logical space used 321090667, configured space 314572800, physical space left 5001654272, partition headroom 10485760, space is not available
traffic_server[57797]: [LOG_FLUSH] NOTE: Cannot clear space because there are no recognized Traffic Server rolled logs for auto deletion.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bneradt commented on issue #7204: Rolled logs are not removed when max_space_mb reached
Posted by GitBox <gi...@apache.org>.
bneradt commented on issue #7204:
URL: https://github.com/apache/trafficserver/issues/7204#issuecomment-698619302
PR is up. Thank you @elsloo for your thorough description of the problem. It helped me reproduce the bug.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bneradt commented on issue #7204: Rolled logs are not removed when max_space_mb reached
Posted by GitBox <gi...@apache.org>.
bneradt commented on issue #7204:
URL: https://github.com/apache/trafficserver/issues/7204#issuecomment-697101517
I've reproduced this in our prod sim environment. After touching `logging.yaml` and performing the `traffic_ctl config reload` as described by Jeff above, Traffic Server seems to lose certain categories of rolled log files it can autodelete. It still can delete rolled traffic.out, diags.out, and manager.out log files, for instance, but not any log files configured via logging.yaml. Once I restart the Traffic Server process, it notices the logging.yaml rolled log files and deletes them, freeing up the space it couldn't before after the config reload.
It's not clear to me why this is happening. I see in the traffic.out output that those logging.yaml configured log files are indeed registered for log deletion, so why aren't they being deleted? I'll look into this more tomorrow morning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org