You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by canselcik <gi...@git.apache.org> on 2016/09/27 01:42:23 UTC
[GitHub] trafficserver pull request #1050: TS-4897: Unbound growth of number of memor...
GitHub user canselcik opened a pull request:
https://github.com/apache/trafficserver/pull/1050
TS-4897: Unbound growth of number of memory maps under SSL termination load when ssl_ticket_enabled=0
The number of `\[anon\]` memory regions mapped to the `traffic_server` process displays unbound growth until the kernel thresholds are reached and the process is terminated.
This happens when ATS is used to terminate SSL and `ssl_ticket_enabled=0` in `ssl_multicert.config`.
We've experienced this issue on our staging and production hosts and were able to replicate it with the above configuration under high volume HTTPS load. We didn't experience this with `5.2.x` and it will make sense why at the end.
While generating `https` traffic with `siege` or `ab`, the issue can be observed with:
`watch "pmap $(pidof traffic_server) | wc -l"`
`git bisect` pointed us to: <TS-3883: Fix madvise>
Turns out a no-op `ats_madvise` hides the symptoms of the issue.
Going in deeper, we realize that `ssl_ticket_enabled` option is relevant because after enabling the `ssl.session_cache` tag, we see that ATS doesn't manage its own session cache for SSL, it is done by the library instead. In that case, the code path doing the problematic allocation within ATS doesn't get executed often since OpenSSL takes care of the session tokens.
But why does this happen? It happens because `MADV_DONTDUMP` is passed to `posix_madvise` even though `MADV_DONTDUMP` is not a valid flag for `posix_madvise` as it is not a drop-in replacement to `madvise`.
Looking at `<bits/mman.h>`:
```
87 /* Advice to `madvise'. */
88 #ifdef __USE_BSD
89 # define MADV_NORMAL\u25b8 0\u25b8 /* No further special treatment. */
90 # define MADV_RANDOM\u25b8 1\u25b8 /* Expect random page references. */
91 # define MADV_SEQUENTIAL 2\u25b8 /* Expect sequential page references. */
92 # define MADV_WILLNEED\u25b8 3\u25b8 /* Will need these pages. */
93 # define MADV_DONTNEED\u25b8 4\u25b8 /* Don't need these pages. */
94 # define MADV_REMOVE\u25b8 9\u25b8 /* Remove these pages and resources. */
95 # define MADV_DONTFORK\u25b8 10\u25b8 /* Do not inherit across fork. */
96 # define MADV_DOFORK\u25b8 11\u25b8 /* Do inherit across fork. */
97 # define MADV_MERGEABLE\u25b8 12\u25b8 /* KSM may merge identical pages. */
98 # define MADV_UNMERGEABLE 13\u25b8 /* KSM may not merge identical pages. */
99 # define MADV_DONTDUMP\u25b8 16 /* Explicity exclude from the core dump,
100 overrides the coredump filter bits. */
101 # define MADV_DODUMP\u25b8 17\u25b8 /* Clear the MADV_DONTDUMP flag. */
102 # define MADV_HWPOISON\u25b8 100\u25b8 /* Poison a page for testing. */
103 #endif
```
However `posix_madvise` takes:
```
107 # define POSIX_MADV_NORMAL\u25b8 0 /* No further special treatment. */
108 # define POSIX_MADV_RANDOM\u25b8 1 /* Expect random page references. */
109 # define POSIX_MADV_SEQUENTIAL\u25b8 2 /* Expect sequential page references. */
110 # define POSIX_MADV_WILLNEED\u25b8 3 /* Will need these pages. */
111 # define POSIX_MADV_DONTNEED\u25b8 4 /* Don't need these pages. */
```
Also `posix_madvise` and `madvise` can both be present on the same system. However they do not have the same capability. That's why `Explicity exclude from the core dump, overrides the coredump filter bits` functionality isn't achievable through `posix_madvise`.
ASF JIRA Reference: https://issues.apache.org/jira/browse/TS-4897
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/canselcik/trafficserver TS4897
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/trafficserver/pull/1050.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1050
----
commit 54df929b630c76990b5b4fcd06b88097088a44e3
Author: Can Selcik <cs...@linkedin.com>
Date: 2016-09-27T01:32:35Z
TS-4897: Unbound growth of number of memory maps for traffic_server under SSL termination load when ssl_ticket_enabled=0
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by smalenfant <gi...@git.apache.org>.
Github user smalenfant commented on the issue:
https://github.com/apache/trafficserver/pull/1050
@PSUdaemon I am yes. Variant of 5.3.2 with the HugePage + madvise patch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by canselcik <gi...@git.apache.org>.
Github user canselcik commented on the issue:
https://github.com/apache/trafficserver/pull/1050
Yeah, I'm starting to get that impression as well.
Observed the issue under:
- 2.6.32-504.el6.x86_64
- 2.6.32-573.7.1.el6.x86_64
Both RHEL so there might be some patches applied to them. I'm looking into getting the source for the build.
Not that `2.6.32` ancient but if this behavior is present with more modern kernel versions, perhaps we could make `MADV_DONTDUMP` behavior configurable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by PSUdaemon <gi...@git.apache.org>.
Github user PSUdaemon commented on the issue:
https://github.com/apache/trafficserver/pull/1050
So it sounds like `madvise()` is a superset of `posix_madvise()` so we should prefer `madvise()` if available? So perhaps we should alter `ats_madvise()` to flip the preference.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by PSUdaemon <gi...@git.apache.org>.
Github user PSUdaemon commented on the issue:
https://github.com/apache/trafficserver/pull/1050
I believe @zwoop has a patch for that somewhere.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver pull request #1050: TS-4897: Unbound growth of number of memor...
Posted by PSUdaemon <gi...@git.apache.org>.
Github user PSUdaemon closed the pull request at:
https://github.com/apache/trafficserver/pull/1050
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by PSUdaemon <gi...@git.apache.org>.
Github user PSUdaemon commented on the issue:
https://github.com/apache/trafficserver/pull/1050
@zwoop: #1097 addresses it as best I think we can.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by jpeach <gi...@git.apache.org>.
Github user jpeach commented on the issue:
https://github.com/apache/trafficserver/pull/1050
See also PR #1097
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by canselcik <gi...@git.apache.org>.
Github user canselcik commented on the issue:
https://github.com/apache/trafficserver/pull/1050
A patch for ATS targeting some issues while running on 2.6.32 or a patch for making `MADV_DONTDUMP` configurable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by jpeach <gi...@git.apache.org>.
Github user jpeach commented on the issue:
https://github.com/apache/trafficserver/pull/1050
@canselcik So if you replace ``posix_madvise`` with ``madvise``, whilst retaining the ``MADV_DONTDUMP`` flag, you observed no increase in memory maps?
One explanation for the difference in memory maps is that ``IOBufferData::alloc()`` does not ``madvise()`` it misses the free list allocated sizes. However these sizes are larger than the lengths in your trace, so that's a conundrum. I think a system call trace might be more illuminating?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by PSUdaemon <gi...@git.apache.org>.
Github user PSUdaemon commented on the issue:
https://github.com/apache/trafficserver/pull/1050
Based on point 3, this sounds like a kernel bug. Which kernel version are you running?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by PSUdaemon <gi...@git.apache.org>.
Github user PSUdaemon commented on the issue:
https://github.com/apache/trafficserver/pull/1050
We feel this has been solved by #1097. Please open a new PR if you feel this is not the case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by jpeach <gi...@git.apache.org>.
Github user jpeach commented on the issue:
https://github.com/apache/trafficserver/pull/1050
FWIW, I have some crash log on an 3.13 (ubuntu) kernel that have an extremely large number of entries in {{/proc/%PID/maps}}. Maybe this problem is not limited to 2.6 kernels?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by canselcik <gi...@git.apache.org>.
Github user canselcik commented on the issue:
https://github.com/apache/trafficserver/pull/1050
I remember seeing the glibc implementation of it basically doing the `madvise` syscall as well but their behavior really do seem to be different on both our production and deployment systems, which are RHEL6.5.
We were also suspecting transparent huge pages but we can elaborate on that potential explanation if this one ends up having shortcomings.
-When `ats_madvice` is a noop, map count doesn't grow unbounded. (regardless of the `ssl_ticket_enabled` value)
-When this change is made and `posix_madvise` is called with `0`, map count doesn't grow unbounded. (regardless of the `ssl_ticket_enabled` value)
-If `ats_madvise` calls `madvise` instead of `posix_madvise` (our systems have both), map count doesn't grow unbounded.
Calls to `ats_madvise` when `ssl_ticket_enabled=0`:
```
ats_madvice(addr=0x7fc4bc022000, len=32768, flags=16) = 0
ats_madvice(addr=0x7fc4b801f000, len=32768, flags=16) = 0
ats_madvice(addr=0x7fc4cc01f000, len=8192, flags=0) = 0
ats_madvice(addr=0x7fc4dc017000, len=32768, flags=16) = 0
ats_madvice(addr=0x7fc4c8005000, len=8192, flags=0) = 0
ats_madvice(addr=0x7fc4d41e5000, len=32768, flags=16) = 0
ats_madvice(addr=0x7fc4a4003000, len=8192, flags=0) = 0
ats_madvice(addr=0x7fc4b0019000, len=32768, flags=16) = 0
ats_madvice(addr=0x7fc4a8013000, len=32768, flags=16) = 0
ats_madvice(addr=0x7fc4a4006000, len=8192, flags=0) = 0
ats_madvice(addr=0x7fc4e0022000, len=32768, flags=16) = 0
ats_madvice(addr=0x1dd7000, len=8192, flags=0) = 0
ats_madvice(addr=0x7fc4c8028000, len=32768, flags=16) = 0
ats_madvice(addr=0x7fc4c408b000, len=32768, flags=16) = 0
ats_madvice(addr=0x7fc4b8003000, len=8192, flags=0) = 0
ats_madvice(addr=0x7fc4b401c000, len=32768, flags=16) = 0
ats_madvice(addr=0x7fc4d801d000, len=8192, flags=0) = 0
ats_madvice(addr=0x7fc4b8028000, len=32768, flags=16) = 0
ats_madvice(addr=0x7fc4b0022000, len=8192, flags=0) = 0
ats_madvice(addr=0x7fc4a401e000, len=32768, flags=16) = 0
ats_madvice(addr=0x7fc4a0003000, len=8192, flags=0) = 0
ats_madvice(addr=0x7fc4d41ee000, len=32768, flags=16) = 0
<pattern repeats as long as more connections are accepted and handed>
```
Calls to `ats_madvise` when `ssl_ticket_enabled=1` (not repeating despite more connections are accepted and handled):
```
ats_madvice(addr=0x7f05aea43000, len=131072, flags=16) = 0
ats_madvice(addr=0x7f057c003000, len=8192, flags=0) = 0
ats_madvice(addr=0x7f057c01e000, len=4096, flags=0) = 0
ats_madvice(addr=0x7f057c022000, len=20480, flags=0) = 0
ats_madvice(addr=0x7f057c028000, len=94208, flags=0) = 0
ats_madvice(addr=0x7f057c040000, len=28672, flags=0) = 0
ats_madvice(addr=0x7f058c02e000, len=32768, flags=0) = 0
ats_madvice(addr=0x7f05aea01000, len=262144, flags=0) = 0
ats_madvice(addr=0x7f05ae9bf000, len=262144, flags=16) = 0
ats_madvice(addr=0x7f05ae97d000, len=262144, flags=0) = 0
ats_madvice(addr=0x7f055c01d000, len=12288, flags=0) = 0
ats_madvice(addr=0x7f05ae93b000, len=262144, flags=16) = 0
ats_madvice(addr=0x7f05ae8f9000, len=262144, flags=16) = 0
ats_madvice(addr=0x7f05ae8b7000, len=262144, flags=16) = 0
ats_madvice(addr=0x7f055c021000, len=12288, flags=0) = 0
```
Even with the non-stop calls to `ats_madvise` in the `ssl_ticket_enabled=0` case, if `ats_madvise` is noop, or uses `flags=0` or calls `madvise` rather than `posix_madvise`, we don't observe the growth in memory maps.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by canselcik <gi...@git.apache.org>.
Github user canselcik commented on the issue:
https://github.com/apache/trafficserver/pull/1050
@jpeach
I've done some further testing (including with strace) and here some corrections and clarifications:
(1) It doesn't matter `ats_madvise` calls `posix_madvise` or `madvise`.
(2) If `posix_madvise` is called with the `MADV_DONTDUMP` flag, it ultimately makes a syscall to `madvise` with `MADV_NORMAL`. It sanitizes the flag to `MADV_NORMAL` if it is greater than 4 basically. In this case we see the growth in memory maps.
(3) If `madvise` is called with the `MADV_DONTDUMP` flag, it ultimately makes a syscall to `madvise` with `MADV_DONTDUMP`, and we see the growth in memory maps.
So basically it looks like you can call `posix_madvise` or `madvise` with `MADV_DONTDUMP` and this issue surfaces.
Any ideas? When we omit the call to either `posix_madvise / madvise` or make the call but make it with `MADV_NORMAL` rather than `MADV_DONTDUMP`, we don't see the issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by smalenfant <gi...@git.apache.org>.
Github user smalenfant commented on the issue:
https://github.com/apache/trafficserver/pull/1050
I've been doing lots of testing and sharing results with @PSUdaemon. I'm seeing the issue when using fairly big ram_cache with the normal pages under Centos 6. I'm not using SSL at all. I've been troubleshooting this since I posted the `ATS blocking at half hour interval` in the traffic server forum.
Test system was multiple Dell R720/730xd with 24 disks and 192GB of memory.
- `CONFIG proxy.config.cache.ram_cache.size INT 103079215104`. => 96GB
Using Centos 6.8 `2.6.32-642.4.2` (version doesn't matter, any variant of Centos 6.7 and 6.8), I would run HLS streaming to fill up the entire ram_cache. At some point (few hours) I would start to see system CPU spikes every 22 minutes. If you run a `perf top` during that time, you can see high activity by `try_to_unmap_anon`.
Then, trying to stop traffic server would take about an hour and there `try_to_unmap_anon` would show up during the entire time.
Now, I then tested with Centos 7 under the same conditions. I didn't see the CPU Spikes and didn't experience the traffic server stop issue.
To make this work with Centos 6, I then configured Huge Pages to stop using anonymous pages. I've set the amount of reserved Huge Pages to 48000 (96GB) and gave it a surplus of 8000 pages (16GB).
```
vm.nr_hugepages = 48000
vm.nr_overcommit_hugepages = 8000
```
You also have to enable Huge Pages `CONFIG proxy.config.allocator.hugepages INT 1`.
Reboot the server to ensure the pages gets allocated without being fragmented (reserved one).
You can then observe the problem is gone (hopefully) and also that the PageTables size is kept to a reasonable size. (Was >200,000kB before).
```
[root@myserver ~]# cat /proc/meminfo | grep Page
AnonPages: 1176352 kB
PageTables: 9836 kB
AnonHugePages: 0 kB
HugePages_Total: 48651
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 651
```
Not sure how related to madvise, but those are the way to get it working and keep performance at a good level. Feel free to comment or ask questions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by PSUdaemon <gi...@git.apache.org>.
Github user PSUdaemon commented on the issue:
https://github.com/apache/trafficserver/pull/1050
This also started appearing once `madvise()` was added. You are using a version of 5.3 with it backported, correct?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by PSUdaemon <gi...@git.apache.org>.
Github user PSUdaemon commented on the issue:
https://github.com/apache/trafficserver/pull/1050
I believe that @smalenfant has seen this same thing and has also observed that upgrading kernels fixes it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by PSUdaemon <gi...@git.apache.org>.
Github user PSUdaemon commented on the issue:
https://github.com/apache/trafficserver/pull/1050
Patch for making the advice configurable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by jpeach <gi...@git.apache.org>.
Github user jpeach commented on the issue:
https://github.com/apache/trafficserver/pull/1050
If you look at the glibc implementation of ``posix_madvise()``, it just makes the ``madvise` system call for the ``MADV_DONTDUMP`` advice.
I assume that you observe the effect when using the ``SSLSessionCache`` implementation of the session cache? What happens when you use ``proxy.config.ssl.session_cache=2``?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by PSUdaemon <gi...@git.apache.org>.
Github user PSUdaemon commented on the issue:
https://github.com/apache/trafficserver/pull/1050
Should we close this now?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1050: TS-4897: Unbound growth of number of memory maps ...
Posted by zwoop <gi...@git.apache.org>.
Github user zwoop commented on the issue:
https://github.com/apache/trafficserver/pull/1050
It's fixed elsewhere?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---