You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by Brian Geffon <br...@gmail.com> on 2019/02/20 02:09:14 UTC
Memory leak at head?
I just pulled down a fresh copy of master, did nothing special:
autoreconf -i
./configure --prefix=/tmp/ats
make -j12 && make install
/tmp/ats/bin/traffic_server
ab -n 10000000 -c 1000 http://127.0.0.1:8080/
I made no changes to any config files other to to enable the memory debug
output every 30 seconds to see where it was going. And the memory stats are
below, but basically it looks like Http1 client session allocators are
leaking pretty badly, I didn't have a chance to dig into it but I wanted to
mention it:
Allocated | In-Use | Type Size | Free List Name
--------------------|--------------------|------------|----------------------------------
0 | 0 | 2097152 |
memory/ioBufAllocator[14]
0 | 0 | 1048576 |
memory/ioBufAllocator[13]
0 | 0 | 524288 |
memory/ioBufAllocator[12]
0 | 0 | 262144 |
memory/ioBufAllocator[11]
0 | 0 | 131072 |
memory/ioBufAllocator[10]
0 | 0 | 65536 |
memory/ioBufAllocator[9]
0 | 0 | 32768 |
memory/ioBufAllocator[8]
0 | 0 | 16384 |
memory/ioBufAllocator[7]
5242880 | 0 | 8192 |
memory/ioBufAllocator[6]
9093251072 | 9091760128 | 4096 |
memory/ioBufAllocator[5]
0 | 0 | 2048 |
memory/ioBufAllocator[4]
0 | 0 | 1024 |
memory/ioBufAllocator[3]
0 | 0 | 512 |
memory/ioBufAllocator[2]
0 | 0 | 256 |
memory/ioBufAllocator[1]
0 | 0 | 128 |
memory/ioBufAllocator[0]
49152 | 3456 | 96 |
memory/eventAllocator
177639120 | 177610240 | 80 |
memory/mutexAllocator
142082048 | 142078912 | 64 |
memory/ioBlockAllocator
106561440 | 106559184 | 48 |
memory/ioDataAllocator
532815360 | 532792800 | 240 | memory/ioAllocator
0 | 0 | 448 |
memory/socksAllocator
0 | 0 | 144 |
memory/udpReadContAllocator
0 | 0 | 160 |
memory/udpPacketAllocator
917504 | 0 | 896 |
memory/netVCAllocator
0 | 0 | 128 |
memory/UDPIOEventAllocator
0 | 0 | 1072 |
memory/sslNetVCAllocator
0 | 0 | 64 |
memory/RamCacheLRUEntry
0 | 0 | 96 |
memory/RamCacheCLFUSEntry
0 | 0 | 160 | memory/openDirEntry
0 | 0 | 48 | memory/evacuationKey
0 | 0 | 64 |
memory/cacheRemoveCont
0 | 0 | 96 |
memory/evacuationBlock
0 | 0 | 960 |
memory/cacheVConnection
0 | 0 | 16 |
memory/DNSRequestDataAllocator
0 | 0 | 29584 |
memory/dnsBufAllocator
0 | 0 | 1296 |
memory/dnsEntryAllocator
0 | 0 | 16 |
memory/expiryQueueEntry
0 | 0 | 64 |
memory/refCountCacheHashingValueAllocator
0 | 0 | 112 |
memory/hostDBFileContAllocator
296960 | 2320 | 2320 |
memory/hostDBContAllocator
0 | 0 | 128 |
memory/OneWayTunnelAllocator
3407872 | 3096576 | 2048 | memory/hdrStrHeap
3670016 | 3096576 | 2048 | memory/hdrHeap
0 | 0 | 256 |
memory/httpCacheAltAllocator
0 | 0 | 3392 |
memory/http2ClientSessionAllocator
0 | 0 | 3120 |
memory/http2StreamAllocator
0 | 0 | 144 |
memory/RemapPluginsAlloc
0 | 0 | 208 |
memory/http1ServerSessionAllocator
13885440 | 0 | 9040 |
memory/httpSMAllocator
2308491120 | 2308454720 | 1040 |
memory/http1ClientSessionAllocator
0 | 0 | 144 |
memory/socksProxyAllocator
0 | 0 | 32 |
memory/MIMEFieldSDKHandle
0 | 0 | 288 |
memory/INKVConnAllocator
0 | 0 | 144 |
memory/INKContAllocator
0 | 0 | 32 |
memory/apiHookAllocator
0 | 0 | 528 |
memory/FetchSMAllocator
1703936 | 0 | 1024 | memory/ArenaBlock
12390013920 | 12365454912 | | TOTAL
Re: Memory leak at head?
Posted by Brian Geffon <br...@gmail.com>.
Done: https://github.com/apache/trafficserver/issues/5042
On Wed, Feb 20, 2019 at 7:20 PM 宋辰伟 <61...@qq.com> wrote:
> Could you guys open issues. It would be more helpful to find the detail
> than email.
>
> Thanks
> Scw00
>
> > 在 2019年2月21日，上午11:13，zzz <zz...@apache.org> 写道：
> >
> > a37f0d91ec0bbc5cd1f0700a69d76df54b9484fc
>
>
Re: Memory leak at head?
Posted by 宋辰伟 <61...@qq.com>.
Could you guys open issues. It would be more helpful to find the detail than email.
Thanks
Scw00
> 在 2019年2月21日，上午11:13，zzz <zz...@apache.org> 写道：
>
> a37f0d91ec0bbc5cd1f0700a69d76df54b9484fc
Re: Memory leak at head?
Posted by zzz <zz...@apache.org>.
git bisect said "a37f0d91ec0bbc5cd1f0700a69d76df54b9484fc is the first bad
commit".
Glad to see Brian here again! We miss you so much! 😘
Brian Geffon <br...@gmail.com> 于2019年2月20日周三 下午4:17写道：
> Lol twist my arm amc ;D
>
> Leif's leak is different that what I'm seeing, mine is trivially reproduced
> but here is the ASAN result:
>
> https://paste.apache.org/6ttx
>
> On Wed, Feb 20, 2019 at 3:55 PM Alan Carroll
> <so...@verizonmedia.com.invalid> wrote:
>
> > I don't think he's "coming" back, as far as I am concerned he's already
> > returned. I think you're onboard with that too, Leif.
> >
> > I've been looking at this more recent leak and I can't figure out what
> > might be going on. It definitely looks like a std::string leak, but I was
> > told the leak is also on master and in that code base, I can't see where
> a
> > std::string would be
> > even created, much less leaked.
> >
> > On Wed, Feb 20, 2019 at 3:43 PM Leif Hedstrom <zw...@apache.org> wrote:
> >
> > >
> > >
> > > > On Feb 20, 2019, at 7:21 AM, Susan Hinrichs <
> shinrich@verizonmedia.com
> > .INVALID>
> > > wrote:
> > > >
> > > > Yes, that does look like a problem. Fei or I will dig into it.
> > >
> > >
> > >
> > > Don’t know if it’s the same, but we’re seeing a pretty significant leak
> > on
> > > Docs / CI:
> > >
> > > https://pastebin.com/jf2WU49i <https://pastebin.com/jf2WU49i>
> > >
> > >
> > > And thanks Brian! When are you coming back to us? ;-)
> > >
> > > Cheers,
> > >
> > > — Leif
> > >
> > >
> >
>
Re: Memory leak at head?
Posted by Brian Geffon <br...@gmail.com>.
Lol twist my arm amc ;D
Leif's leak is different that what I'm seeing, mine is trivially reproduced
but here is the ASAN result:
https://paste.apache.org/6ttx
On Wed, Feb 20, 2019 at 3:55 PM Alan Carroll
<so...@verizonmedia.com.invalid> wrote:
> I don't think he's "coming" back, as far as I am concerned he's already
> returned. I think you're onboard with that too, Leif.
>
> I've been looking at this more recent leak and I can't figure out what
> might be going on. It definitely looks like a std::string leak, but I was
> told the leak is also on master and in that code base, I can't see where a
> std::string would be
> even created, much less leaked.
>
> On Wed, Feb 20, 2019 at 3:43 PM Leif Hedstrom <zw...@apache.org> wrote:
>
> >
> >
> > > On Feb 20, 2019, at 7:21 AM, Susan Hinrichs <shinrich@verizonmedia.com
> .INVALID>
> > wrote:
> > >
> > > Yes, that does look like a problem. Fei or I will dig into it.
> >
> >
> >
> > Don’t know if it’s the same, but we’re seeing a pretty significant leak
> on
> > Docs / CI:
> >
> > https://pastebin.com/jf2WU49i <https://pastebin.com/jf2WU49i>
> >
> >
> > And thanks Brian! When are you coming back to us? ;-)
> >
> > Cheers,
> >
> > — Leif
> >
> >
>
Re: Memory leak at head?
Posted by Alan Carroll <so...@verizonmedia.com.INVALID>.
I don't think he's "coming" back, as far as I am concerned he's already
returned. I think you're onboard with that too, Leif.
I've been looking at this more recent leak and I can't figure out what
might be going on. It definitely looks like a std::string leak, but I was
told the leak is also on master and in that code base, I can't see where a
std::string would be
even created, much less leaked.
On Wed, Feb 20, 2019 at 3:43 PM Leif Hedstrom <zw...@apache.org> wrote:
>
>
> > On Feb 20, 2019, at 7:21 AM, Susan Hinrichs <sh...@verizonmedia.com.INVALID>
> wrote:
> >
> > Yes, that does look like a problem. Fei or I will dig into it.
>
>
>
> Don’t know if it’s the same, but we’re seeing a pretty significant leak on
> Docs / CI:
>
> https://pastebin.com/jf2WU49i <https://pastebin.com/jf2WU49i>
>
>
> And thanks Brian! When are you coming back to us? ;-)
>
> Cheers,
>
> — Leif
>
>
Re: Memory leak at head?
Posted by Leif Hedstrom <zw...@apache.org>.
> On Feb 20, 2019, at 7:21 AM, Susan Hinrichs <sh...@verizonmedia.com.INVALID> wrote:
>
> Yes, that does look like a problem. Fei or I will dig into it.
Don’t know if it’s the same, but we’re seeing a pretty significant leak on Docs / CI:
https://pastebin.com/jf2WU49i <https://pastebin.com/jf2WU49i>
And thanks Brian! When are you coming back to us? ;-)
Cheers,
— Leif
Re: Memory leak at head?
Posted by Susan Hinrichs <sh...@verizonmedia.com.INVALID>.
Yes, that does look like a problem. Fei or I will dig into it.
On Tue, Feb 19, 2019 at 8:09 PM Brian Geffon <br...@gmail.com> wrote:
> I just pulled down a fresh copy of master, did nothing special:
>
> autoreconf -i
> ./configure --prefix=/tmp/ats
> make -j12 && make install
>
> /tmp/ats/bin/traffic_server
> ab -n 10000000 -c 1000 http://127.0.0.1:8080/
>
> I made no changes to any config files other to to enable the memory debug
> output every 30 seconds to see where it was going. And the memory stats are
> below, but basically it looks like Http1 client session allocators are
> leaking pretty badly, I didn't have a chance to dig into it but I wanted to
> mention it:
>
> Allocated | In-Use | Type Size | Free List Name
>
> --------------------|--------------------|------------|----------------------------------
> 0 | 0 | 2097152 |
> memory/ioBufAllocator[14]
> 0 | 0 | 1048576 |
> memory/ioBufAllocator[13]
> 0 | 0 | 524288 |
> memory/ioBufAllocator[12]
> 0 | 0 | 262144 |
> memory/ioBufAllocator[11]
> 0 | 0 | 131072 |
> memory/ioBufAllocator[10]
> 0 | 0 | 65536 |
> memory/ioBufAllocator[9]
> 0 | 0 | 32768 |
> memory/ioBufAllocator[8]
> 0 | 0 | 16384 |
> memory/ioBufAllocator[7]
> 5242880 | 0 | 8192 |
> memory/ioBufAllocator[6]
> 9093251072 | 9091760128 | 4096 |
> memory/ioBufAllocator[5]
> 0 | 0 | 2048 |
> memory/ioBufAllocator[4]
> 0 | 0 | 1024 |
> memory/ioBufAllocator[3]
> 0 | 0 | 512 |
> memory/ioBufAllocator[2]
> 0 | 0 | 256 |
> memory/ioBufAllocator[1]
> 0 | 0 | 128 |
> memory/ioBufAllocator[0]
> 49152 | 3456 | 96 |
> memory/eventAllocator
> 177639120 | 177610240 | 80 |
> memory/mutexAllocator
> 142082048 | 142078912 | 64 |
> memory/ioBlockAllocator
> 106561440 | 106559184 | 48 |
> memory/ioDataAllocator
> 532815360 | 532792800 | 240 | memory/ioAllocator
> 0 | 0 | 448 |
> memory/socksAllocator
> 0 | 0 | 144 |
> memory/udpReadContAllocator
> 0 | 0 | 160 |
> memory/udpPacketAllocator
> 917504 | 0 | 896 |
> memory/netVCAllocator
> 0 | 0 | 128 |
> memory/UDPIOEventAllocator
> 0 | 0 | 1072 |
> memory/sslNetVCAllocator
> 0 | 0 | 64 |
> memory/RamCacheLRUEntry
> 0 | 0 | 96 |
> memory/RamCacheCLFUSEntry
> 0 | 0 | 160 | memory/openDirEntry
> 0 | 0 | 48 |
> memory/evacuationKey
> 0 | 0 | 64 |
> memory/cacheRemoveCont
> 0 | 0 | 96 |
> memory/evacuationBlock
> 0 | 0 | 960 |
> memory/cacheVConnection
> 0 | 0 | 16 |
> memory/DNSRequestDataAllocator
> 0 | 0 | 29584 |
> memory/dnsBufAllocator
> 0 | 0 | 1296 |
> memory/dnsEntryAllocator
> 0 | 0 | 16 |
> memory/expiryQueueEntry
> 0 | 0 | 64 |
> memory/refCountCacheHashingValueAllocator
> 0 | 0 | 112 |
> memory/hostDBFileContAllocator
> 296960 | 2320 | 2320 |
> memory/hostDBContAllocator
> 0 | 0 | 128 |
> memory/OneWayTunnelAllocator
> 3407872 | 3096576 | 2048 | memory/hdrStrHeap
> 3670016 | 3096576 | 2048 | memory/hdrHeap
> 0 | 0 | 256 |
> memory/httpCacheAltAllocator
> 0 | 0 | 3392 |
> memory/http2ClientSessionAllocator
> 0 | 0 | 3120 |
> memory/http2StreamAllocator
> 0 | 0 | 144 |
> memory/RemapPluginsAlloc
> 0 | 0 | 208 |
> memory/http1ServerSessionAllocator
> 13885440 | 0 | 9040 |
> memory/httpSMAllocator
> 2308491120 | 2308454720 | 1040 |
> memory/http1ClientSessionAllocator
> 0 | 0 | 144 |
> memory/socksProxyAllocator
> 0 | 0 | 32 |
> memory/MIMEFieldSDKHandle
> 0 | 0 | 288 |
> memory/INKVConnAllocator
> 0 | 0 | 144 |
> memory/INKContAllocator
> 0 | 0 | 32 |
> memory/apiHookAllocator
> 0 | 0 | 528 |
> memory/FetchSMAllocator
> 1703936 | 0 | 1024 | memory/ArenaBlock
> 12390013920 | 12365454912 | | TOTAL
>