[jira] [Comment Edited] (TS-3554) ATS memory leak reloading ssl_multicert.config with many ssl cert configs

["Brian Geffon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TS-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14547470#comment-14547470 ] 

Brian Geffon edited comment on TS-3554 at 5/18/15 2:49 AM:
-----------------------------------------------------------

[~shinrich] your analysis seems correct; however, I think we need to fully blow away the session cache on reinitialize, it's not enough to have the session cache size change because if they keys change you also need to invalidate sessions so I think the safest way to go is in the event of a reinitialization to just free an existing session cache.

I do think we need to be careful w.r.t the shared data structure. We'll need to ref count the parent session_cache object, luckily the count can be decremented after a session is established but we'll need to make sure the ssl, perhaps r/w locks would make the most sense here? If we take a r/w lock approach we can take a write lock whenever we want to swap out the existing structure so we can wait for all "readers" to finish and block new ones while we're waiting to swap out the new session cache...thoughts?

One scenario I was worried about was where we try to add a session that was established using an old key or something into the swapped cache:

Thread 1: establish new ssl session (S) when ssl session cache is version X
Thread 2: swap out ssl keys and ssl session cache to version Y
Thread 1: open ssl calls back to add ssl session (S) to new version of cache Y

Obviously that's probably not the intended behavior...so i'm not really sure how this should be reconciled.


was (Author: briang):
[~shinrich] your analysis seems correct; however, I think we need to fully blow away the session cache on reinitialize, it's not enough to have the session cache size change because if they keys change you also need to invalidate sessions so I think the safest way to go is in the event of a reinitialization to just free an existing session cache.

I do think we need to be careful w.r.t the shared data structure. We'll need to ref count the parent session_cache object, luckily the count can be decremented after a session is established but we'll need to make sure the ssl, perhaps r/w locks would make the most sense here? If we take a r/w lock approach we can take a write lock whenever we want to swap out the existing structure so we can wait for all "readers" to finish and block new ones while we're waiting to swap out the new session cache...thoughts?

> ATS memory leak reloading ssl_multicert.config with many ssl cert configs
> -------------------------------------------------------------------------
>
>                 Key: TS-3554
>                 URL: https://issues.apache.org/jira/browse/TS-3554
>             Project: Traffic Server
>          Issue Type: Bug
>          Components: Configuration, Core, SSL
>            Reporter: Steven Feltner
>            Assignee: Susan Hinrichs
>             Fix For: 6.0.0
>
>         Attachments: limit_session_cache_alloc.diff, ts-3554-53-2.diff, ts-3554-53.diff
>
>
> ATS will consume all available memory on a server with 128GB of RAM.  @shinrich suspects it may be due to CertLookup table not being freed on a config reload.
> Our current process:
> - New cert comes in
> - ssl_multicert.config and remap.config updated
> - traffic_line -x
> This reload could occur as often as every 3 mins with 5000+ certs configured.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)