You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Steve Malenfant <sm...@gmail.com> on 2016/11/01 16:35:10 UTC
Corrupted cache? Looping on directory cleaning.
We have a server which had a bad disk (out of 24). Trafficserver was
stopped and restarted before the disk was swapped (the disk has been marked
offline for a week). This is on version is 5.3.2 with some patches.
Stop, cleared the cache and restarted. Problem is gone.
Once it was restarted, I kept seeing this with "perf top" and CPU usage is
100% with load average of 1. Every request is a MISS.
Anybody seen this before?
Been going for more than 24 hours and server has been idle :
50.74% traffic_server [.] dir_clean_segment
30.65% traffic_server [.] dir_delete_entry
If I look a disk IO, there is nothing going on. Looks like stuck in a loop.
Enabling debug produced the following :
[Nov 1 16:06:31.127] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning
0x2b372a76a838 tag 1A boffset 0 b 0x2b372a76a838 p (nil) l -1
[Nov 1 16:06:31.127] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning
0x2b372a76a838 tag 1E boffset 0 b 0x2b372a76a838 p (nil) l -1
[Nov 1 16:06:31.127] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning
0x2b372a76a838 tag 2E boffset 0 b 0x2b372a76a838 p (nil) l -1
[Nov 1 16:06:31.127] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning
0x2b372a76a838 tag 32 boffset 0 b 0x2b372a76a838 p (nil) l -1
[Nov 1 16:06:31.127] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning
0x2b372a76a838 tag 3A boffset 0 b 0x2b372a76a838 p (nil) l -1
[Nov 1 16:06:31.127] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning
0x2b372a76a838 tag 3E boffset 0 b 0x2b372a76a838 p (nil) l -1
[Nov 1 16:06:31.127] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning
0x2b372a76a838 tag 42 boffset 0 b 0x2b372a76a838 p (nil) l -1
[Nov 1 16:18:31.102] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning
0x2b372a76a838 tag 171 boffset 0 b 0x2b372a76a838 p (nil) l -1
Re: Corrupted cache? Looping on directory cleaning.
Posted by Alan Carroll <so...@yahoo-inc.com>.
Hmmm. I think that's too old for using the stripe inspector. There's at least one JIRA ticket on this already, at this point unfortunately I think your only option is to clear the cache, although you said you had already done that? On the other hand it shouldn't be cleaning anything if the cache has been cleared.
On Tuesday, November 1, 2016 11:35 AM, Steve Malenfant <sm...@gmail.com> wrote:
We have a server which had a bad disk (out of 24). Trafficserver was stopped and restarted before the disk was swapped (the disk has been marked offline for a week). This is on version is 5.3.2 with some patches.
Stop, cleared the cache and restarted. Problem is gone.
Once it was restarted, I kept seeing this with "perf top" and CPU usage is 100% with load average of 1. Every request is a MISS.
Anybody seen this before?
Been going for more than 24 hours and server has been idle :
50.74% traffic_server [.] dir_clean_segment 30.65% traffic_server [.] dir_delete_entry
If I look a disk IO, there is nothing going on. Looks like stuck in a loop. Enabling debug produced the following :[Nov 1 16:06:31.127] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning 0x2b372a76a838 tag 1A boffset 0 b 0x2b372a76a838 p (nil) l -1[Nov 1 16:06:31.127] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning 0x2b372a76a838 tag 1E boffset 0 b 0x2b372a76a838 p (nil) l -1[Nov 1 16:06:31.127] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning 0x2b372a76a838 tag 2E boffset 0 b 0x2b372a76a838 p (nil) l -1[Nov 1 16:06:31.127] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning 0x2b372a76a838 tag 32 boffset 0 b 0x2b372a76a838 p (nil) l -1[Nov 1 16:06:31.127] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning 0x2b372a76a838 tag 3A boffset 0 b 0x2b372a76a838 p (nil) l -1[Nov 1 16:06:31.127] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning 0x2b372a76a838 tag 3E boffset 0 b 0x2b372a76a838 p (nil) l -1[Nov 1 16:06:31.127] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning 0x2b372a76a838 tag 42 boffset 0 b 0x2b372a76a838 p (nil) l -1[Nov 1 16:18:31.102] Server {0x2b305e827700} DEBUG: (dir_clean) cleaning 0x2b372a76a838 tag 171 boffset 0 b 0x2b372a76a838 p (nil) l -1