You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Ruediger Pluem <rp...@apache.org> on 2019/02/08 07:07:57 UTC
Re: svn commit: r1841225 -
/httpd/httpd/trunk/modules/dav/main/props.c
On 11/13/2018 10:26 AM, Joe Orton wrote:
> On Mon, Nov 12, 2018 at 08:26:48AM +0100, Ruediger Pluem wrote:
>> The discussion died a little bit, because of the other issue (frequent
>> writeev calls). I know that the liveprops issue is not fixed yet, but
>> I guess it makes sense if you commit the patch you posted here
>> already.
>
> Sorry, I was looking at this again and the repro case I thought worked
> was still showing leaks, so I got stuck. Will come back to it hopefully
> later this week.
>
Going through the STATUS of 2.4.x I got aware that this died a little bit again.
Any new ideas in the meantime?
Regards
Rüdiger
AW: svn commit: r1841225 -
/httpd/httpd/trunk/modules/dav/main/props.c
Posted by Plüm,
Rüdiger,
Vodafone Group <ru...@vodafone.com>.
C2 General
> -----Ursprüngliche Nachricht-----
> Von: Joe Orton <jo...@redhat.com>
> Gesendet: Dienstag, 25. Juni 2019 13:59
> An: dev@httpd.apache.org
> Betreff: Re: svn commit: r1841225 -
> /httpd/httpd/trunk/modules/dav/main/props.c
>
> On Tue, Jun 25, 2019 at 09:06:42AM +0000, Plüm, Rüdiger, Vodafone Group
> wrote:
> > > On Tue, Jun 25, 2019 at 08:49:08AM +0100, Joe Orton wrote:
> > > > Unless I am missing something the apr_dir_open/apr_dir_read API
> design
> > > > is always going to have memory use proportional to directory size
> > > > because apr_dir_read() duplicates the filename into the
> directory's
> > > > pool. We need an apr_dir_pread() or whatever which takes a pool
> > > > argument. :(
> > >
> > > OK, yes, this plus an iterpool in dav_fs_walker() fixes it.
> >
> > Only the iterpool in dav_fs_walker or plus the changes in APR
> (apr_dir_pread)?
>
> Yes, both - PoC attached but I'm not at all confident this is safe, esp
> in the recursion case.
>
> The test I was running before was only a "core" liveprop provided by
> mod_dav itself, so I also tested the deadprops case and that still seems
> to leak. sbrk() to increase the heap is getting triggered by
> apr_sdbm_open() every time I saw it, but apr_dbm_open() is being pass
> the scratchpool so I think it's not the cause.
I would agree, but can you please provide the stacktrace up to sbrk?
That would ease it for me to dive in :-).
>
> Running "dump_all_pools propdb->p" at this point gives me:
>
> Pool 'transaction' [0x24e1d88]: 7264/8192 free (1 blocks) allocator:
> 0x24d2b90 free blocks in allocator: 0 kiB
> Pool 'master_conn' [0x2504e28]: 1224/8192 free (1 blocks)
> allocator: 0x24d2b90 free blocks in allocator: 0 kiB
> Pool 'deferred_pool' [0x263ef18]: 8032/8192 free (1 blocks)
> allocator: 0x24d2b90 free blocks in allocator: 0 kiB
> Pool 'request' [0x2506e38]: 13672/1646592 free (201 blocks)
> allocator: 0x24d2b90 free blocks in allocator: 0 kiB
> Pool 'mod_dav_fs-walker' [0x2524648]: 8016/8192 free (1
> blocks) allocator: 0x24d2b90 free blocks in allocator: 0 kiB
> Pool 'mod_dav-scratch' [0x2508e48]: 7224/8192 free (1 blocks)
> allocator: 0x24d2b90 free blocks in allocator: 0 kiB
> Pool 'mod_lua-vm' [0x250ce68]: 6360/16384 free (2 blocks)
> allocator: 0x24d2b90 free blocks in allocator: 0 kiB
>
> [trimmed some whitespace only]
>
> I think this is implicating misuse of r->pool, that's grown rather
> large?
>
I would think so as well.
Regards
Rüdiger
Re: svn commit: r1841225 -
/httpd/httpd/trunk/modules/dav/main/props.c
Posted by Joe Orton <jo...@redhat.com>.
On Tue, Jun 25, 2019 at 09:06:42AM +0000, Plüm, Rüdiger, Vodafone Group wrote:
> > On Tue, Jun 25, 2019 at 08:49:08AM +0100, Joe Orton wrote:
> > > Unless I am missing something the apr_dir_open/apr_dir_read API design
> > > is always going to have memory use proportional to directory size
> > > because apr_dir_read() duplicates the filename into the directory's
> > > pool. We need an apr_dir_pread() or whatever which takes a pool
> > > argument. :(
> >
> > OK, yes, this plus an iterpool in dav_fs_walker() fixes it.
>
> Only the iterpool in dav_fs_walker or plus the changes in APR (apr_dir_pread)?
Yes, both - PoC attached but I'm not at all confident this is safe, esp
in the recursion case.
The test I was running before was only a "core" liveprop provided by
mod_dav itself, so I also tested the deadprops case and that still seems
to leak. sbrk() to increase the heap is getting triggered by
apr_sdbm_open() every time I saw it, but apr_dbm_open() is being pass
the scratchpool so I think it's not the cause.
Running "dump_all_pools propdb->p" at this point gives me:
Pool 'transaction' [0x24e1d88]: 7264/8192 free (1 blocks) allocator: 0x24d2b90 free blocks in allocator: 0 kiB
Pool 'master_conn' [0x2504e28]: 1224/8192 free (1 blocks) allocator: 0x24d2b90 free blocks in allocator: 0 kiB
Pool 'deferred_pool' [0x263ef18]: 8032/8192 free (1 blocks) allocator: 0x24d2b90 free blocks in allocator: 0 kiB
Pool 'request' [0x2506e38]: 13672/1646592 free (201 blocks) allocator: 0x24d2b90 free blocks in allocator: 0 kiB
Pool 'mod_dav_fs-walker' [0x2524648]: 8016/8192 free (1 blocks) allocator: 0x24d2b90 free blocks in allocator: 0 kiB
Pool 'mod_dav-scratch' [0x2508e48]: 7224/8192 free (1 blocks) allocator: 0x24d2b90 free blocks in allocator: 0 kiB
Pool 'mod_lua-vm' [0x250ce68]: 6360/16384 free (2 blocks) allocator: 0x24d2b90 free blocks in allocator: 0 kiB
[trimmed some whitespace only]
I think this is implicating misuse of r->pool, that's grown rather
large?
Regards, Joe
AW: svn commit: r1841225 -
/httpd/httpd/trunk/modules/dav/main/props.c
Posted by Plüm,
Rüdiger,
Vodafone Group <ru...@vodafone.com>.
C2 General
> -----Ursprüngliche Nachricht-----
> Von: Joe Orton <jo...@redhat.com>
> Gesendet: Dienstag, 25. Juni 2019 10:45
> An: dev@httpd.apache.org
> Betreff: Re: svn commit: r1841225 -
> /httpd/httpd/trunk/modules/dav/main/props.c
>
> On Tue, Jun 25, 2019 at 08:49:08AM +0100, Joe Orton wrote:
> > Unless I am missing something the apr_dir_open/apr_dir_read API design
> > is always going to have memory use proportional to directory size
> > because apr_dir_read() duplicates the filename into the directory's
> > pool. We need an apr_dir_pread() or whatever which takes a pool
> > argument. :(
>
> OK, yes, this plus an iterpool in dav_fs_walker() fixes it.
Only the iterpool in dav_fs_walker or plus the changes in APR (apr_dir_pread)?
>
> I've been using the python "psrecord" (from pip) to check this rather
> than my traditional method of "eyeballing top", here are the charts for
> running 10x a PROPFIND with ~100K files, for
Great plots. Thanks for pointing to the tool.
>
> a) trunk - steep gradient, ~20mb consumed
> b) trunk plus the patch from my previous post, ~5mb?
> c) (b) plus iterpool in dav_fs_walker - flat
>
> so I think (b) is fine to commit but with the recursion in dav_fs_walker
> I am not at all sure this stuff is safe yet, will need some more
> testing.
The recursion stuff could be tricky. Probably a separate (sub-)pool for apr_dir_open
on each recursion level and separate iterpools on each recursion level?
Regards
Rüdiger
Re: svn commit: r1841225 -
/httpd/httpd/trunk/modules/dav/main/props.c
Posted by Stefan Eissing <st...@greenbytes.de>.
Nice work!
> Am 25.06.2019 um 10:44 schrieb Joe Orton <jo...@redhat.com>:
>
> <ps-trunk.png>
Re: svn commit: r1841225 -
/httpd/httpd/trunk/modules/dav/main/props.c
Posted by Joe Orton <jo...@redhat.com>.
On Tue, Jun 25, 2019 at 08:49:08AM +0100, Joe Orton wrote:
> Unless I am missing something the apr_dir_open/apr_dir_read API design
> is always going to have memory use proportional to directory size
> because apr_dir_read() duplicates the filename into the directory's
> pool. We need an apr_dir_pread() or whatever which takes a pool
> argument. :(
OK, yes, this plus an iterpool in dav_fs_walker() fixes it.
I've been using the python "psrecord" (from pip) to check this rather
than my traditional method of "eyeballing top", here are the charts for
running 10x a PROPFIND with ~100K files, for
a) trunk - steep gradient, ~20mb consumed
b) trunk plus the patch from my previous post, ~5mb?
c) (b) plus iterpool in dav_fs_walker - flat
so I think (b) is fine to commit but with the recursion in dav_fs_walker
I am not at all sure this stuff is safe yet, will need some more
testing.
Re: svn commit: r1841225 -
/httpd/httpd/trunk/modules/dav/main/props.c
Posted by Ruediger Pluem <rp...@apache.org>.
On 06/24/2019 07:04 PM, Joe Orton wrote:
> On Fri, Feb 08, 2019 at 08:07:57AM +0100, Ruediger Pluem wrote:
>> On 11/13/2018 10:26 AM, Joe Orton wrote:
>>> On Mon, Nov 12, 2018 at 08:26:48AM +0100, Ruediger Pluem wrote:
>>>> The discussion died a little bit, because of the other issue (frequent
>>>> writeev calls). I know that the liveprops issue is not fixed yet, but
>>>> I guess it makes sense if you commit the patch you posted here
>>>> already.
>>>
>>> Sorry, I was looking at this again and the repro case I thought worked
>>> was still showing leaks, so I got stuck. Will come back to it hopefully
>>> later this week.
>>>
>>
>> Going through the STATUS of 2.4.x I got aware that this died a little bit again.
>> Any new ideas in the meantime?
>
> I spent another half an afternoon looking at this.
>
> I'm trying a PROPFIND/depth: 1 for *just* DAV: getcontenttype - and I
> still see steadily increasing memory use during the response with trunk
> or trunk plus my patch.
>
> If I break in sbrk the memory allocation which triggers a heap expansion
> is within some r->pool for the subrequest, so I suppose the bug is
> around subreq handling somehow, but it's far from obvious.
Thanks for spending time on this again. By coincidence I stumbled across it
recently again on a system that has Webdav enabled folders with a lot of files in (e.g. 10,000).
Over a short time the only process of this webserver instance with 10 threads, which does
nothing else but serving this stuff via WebDav rose to several GB.
Doing a dump_all_pools showed that all pools and allocator free lists looked well (MaxMemFree
set to 2048 globally).
I was able to limit the memory consumption over time to sane values by adding the following environment
variables:
export MALLOC_MMAP_THRESHOLD_=8192
export MALLOC_ARENA_MAX=3
The system is running RedHat 7.5. So my conclusion was that the glibc was not able to free the memory
again when it was allocated via sbrk, probably because other blocks that remained in use were added
to the top of the heap. OTOH memory also does not seem to get reused with the next request which
is kind of weird given that APR requests pages size aligned blocks which should reduce fragmentation.
Regards
Rüdiger
Re: AW: svn commit: r1841225 -
/httpd/httpd/trunk/modules/dav/main/props.c
Posted by Ruediger Pluem <rp...@apache.org>.
On 06/27/2019 10:45 PM, William A Rowe Jr wrote:
> On Thu, Jun 27, 2019 at 2:06 PM Ruediger Pluem <rpluem@apache.org <ma...@apache.org>> wrote:
>
>
> On 06/25/2019 10:39 AM, Plüm, Rüdiger, Vodafone Group wrote:
> >
> > Another related question: Do you think it would be beneficial if we replace
> > the apr_psprintf with apr_pstrcat in props.c as long as they only concat strings without
> > further formatting?
>
> Anyone any opinion on this?
>
>
> I would expect apr_pstrcat to be more performant, wise change.
Done in r1862270.
Regards
Rüdiger
Re: AW: svn commit: r1841225 - /httpd/httpd/trunk/modules/dav/main/props.c
Posted by William A Rowe Jr <wr...@rowe-clan.net>.
On Thu, Jun 27, 2019 at 2:06 PM Ruediger Pluem <rp...@apache.org> wrote:
>
> On 06/25/2019 10:39 AM, Plüm, Rüdiger, Vodafone Group wrote:
> >
> > Another related question: Do you think it would be beneficial if we
> replace
> > the apr_psprintf with apr_pstrcat in props.c as long as they only concat
> strings without
> > further formatting?
>
> Anyone any opinion on this?
>
I would expect apr_pstrcat to be more performant, wise change.
Re: AW: svn commit: r1841225 -
/httpd/httpd/trunk/modules/dav/main/props.c
Posted by Ruediger Pluem <rp...@apache.org>.
On 06/25/2019 10:39 AM, Plüm, Rüdiger, Vodafone Group wrote:
>
>
>
>
> Another related question: Do you think it would be beneficial if we replace
> the apr_psprintf with apr_pstrcat in props.c as long as they only concat strings without
> further formatting?
>
>
Anyone any opinion on this?
Regards
Rüdiger
AW: svn commit: r1841225 -
/httpd/httpd/trunk/modules/dav/main/props.c
Posted by Plüm,
Rüdiger,
Vodafone Group <ru...@vodafone.com>.
C2 General
> -----Ursprüngliche Nachricht-----
> Von: Joe Orton <jo...@redhat.com>
> Gesendet: Dienstag, 25. Juni 2019 09:49
> An: dev@httpd.apache.org
> Betreff: Re: svn commit: r1841225 -
> /httpd/httpd/trunk/modules/dav/main/props.c
>
> On Mon, Jun 24, 2019 at 09:54:21PM +0200, Ruediger Pluem wrote:
> > On 06/24/2019 07:04 PM, Joe Orton wrote:
> > > #11 0x00007f8110ce6448 in dav_do_prop_subreq (propdb=0x100cfc0) at
> props.c:331
> >
> > What if you go here and do a
> >
> > dump_all_pools propdb->p
> >
> > and do it again when you hit the next sbrk and the next one and so on?
> Something suspicious?
>
> Well I did this and immediately noticed something suspicious on the
> previous line, e_uri is allocated from resource->pool (=r->pool) rather
> than propbb->p, so, yet again the precision accuracy of your insights is
Very good catch. Great that you found it.
> beyond belief Ruediger :) :)
>
> This fixed a significant leak, but r->pool is still growing. I think
> the remaining problem is use of r->pool in dav_fs_walker, i.e. is
> mod_dav_fs specific.
Do you think it is now good enough to commit at least as a first step?
>
> Unless I am missing something the apr_dir_open/apr_dir_read API design
> is always going to have memory use proportional to directory size
> because apr_dir_read() duplicates the filename into the directory's
> pool. We need an apr_dir_pread() or whatever which takes a pool
> argument. :(
In order to fix this leak, I agree that we would need this.
The pool is also used for apr_stat inside apr_dir_read, but I fail to see
how it consumes memory.
Another related question: Do you think it would be beneficial if we replace
the apr_psprintf with apr_pstrcat in props.c as long as they only concat strings without
further formatting?
Regards
Rüdiger
Re: svn commit: r1841225 -
/httpd/httpd/trunk/modules/dav/main/props.c
Posted by Joe Orton <jo...@redhat.com>.
On Mon, Jun 24, 2019 at 09:54:21PM +0200, Ruediger Pluem wrote:
> On 06/24/2019 07:04 PM, Joe Orton wrote:
> > #11 0x00007f8110ce6448 in dav_do_prop_subreq (propdb=0x100cfc0) at props.c:331
>
> What if you go here and do a
>
> dump_all_pools propdb->p
>
> and do it again when you hit the next sbrk and the next one and so on? Something suspicious?
Well I did this and immediately noticed something suspicious on the
previous line, e_uri is allocated from resource->pool (=r->pool) rather
than propbb->p, so, yet again the precision accuracy of your insights is
beyond belief Ruediger :) :)
This fixed a significant leak, but r->pool is still growing. I think
the remaining problem is use of r->pool in dav_fs_walker, i.e. is
mod_dav_fs specific.
Unless I am missing something the apr_dir_open/apr_dir_read API design
is always going to have memory use proportional to directory size
because apr_dir_read() duplicates the filename into the directory's
pool. We need an apr_dir_pread() or whatever which takes a pool
argument. :(
Re: svn commit: r1841225 -
/httpd/httpd/trunk/modules/dav/main/props.c
Posted by Ruediger Pluem <rp...@apache.org>.
On 06/24/2019 07:04 PM, Joe Orton wrote:
> On Fri, Feb 08, 2019 at 08:07:57AM +0100, Ruediger Pluem wrote:
>> On 11/13/2018 10:26 AM, Joe Orton wrote:
>>> On Mon, Nov 12, 2018 at 08:26:48AM +0100, Ruediger Pluem wrote:
>>>> The discussion died a little bit, because of the other issue (frequent
>>>> writeev calls). I know that the liveprops issue is not fixed yet, but
>>>> I guess it makes sense if you commit the patch you posted here
>>>> already.
>>>
>>> Sorry, I was looking at this again and the repro case I thought worked
>>> was still showing leaks, so I got stuck. Will come back to it hopefully
>>> later this week.
>>>
>>
>> Going through the STATUS of 2.4.x I got aware that this died a little bit again.
>> Any new ideas in the meantime?
>
>
> If I break in sbrk the memory allocation which triggers a heap expansion
> is within some r->pool for the subrequest, so I suppose the bug is
> around subreq handling somehow, but it's far from obvious.
>
> #0 0x00007f8111b60130 in sbrk () from /lib64/libc.so.6
> #1 0x00007f8111af54cd in __default_morecore () from /lib64/libc.so.6
> #2 0x00007f8111af18a3 in sysmalloc () from /lib64/libc.so.6
> #3 0x00007f8111af2a3f in _int_malloc () from /lib64/libc.so.6
> #4 0x00007f8111af3b8f in malloc () from /lib64/libc.so.6
> #5 0x00007f8111c7cd34 in allocator_alloc (in_size=8152, allocator=0xfd69f0) at memory/unix/apr_pools.c:411
> #6 apr_pool_create_ex (newpool=newpool@entry=0x7ffe8dc303f8, parent=0x16fe6b8, abort_fn=0x440d90 <abort_on_oom>,
> abort_fn@entry=0x0, allocator=0xfd69f0, allocator@entry=0x0) at memory/unix/apr_pools.c:1079
> #7 0x0000000000461a4a in ap_location_walk (r=r@entry=0x16fe730) at request.c:1487
> #8 0x0000000000462164 in ap_process_request_internal (r=0x16fe730) at request.c:238
> #9 0x0000000000462cc8 in ap_sub_req_method_uri (method=method@entry=0x47a6a0 "GET",
> new_uri=0x16fc6a8 "/modules/dav/file.b626.txt", r=0x100afb0, next_filter=next_filter@entry=0x0) at request.c:2346
> #10 0x0000000000462d03 in ap_sub_req_lookup_uri (new_uri=<optimized out>, r=<optimized out>,
> next_filter=next_filter@entry=0x0) at request.c:2359
> #11 0x00007f8110ce6448 in dav_do_prop_subreq (propdb=0x100cfc0) at props.c:331
What if you go here and do a
dump_all_pools propdb->p
and do it again when you hit the next sbrk and the next one and so on? Something suspicious?
> #12 0x00007f8110ce665d in dav_insert_coreprop (propdb=propdb@entry=0x100cfc0, propid=<optimized out>,
> name=0x1014e18 "getcontenttype", what=what@entry=DAV_PROP_INSERT_VALUE, phdr=phdr@entry=0x7ffe8dc30550,
> inserted=inserted@entry=0x7ffe8dc30548) at props.c:390
> #13 0x00007f8110ce73b5 in dav_insert_liveprop (what=DAV_PROP_INSERT_VALUE, elem=0x1014da8, elem=0x1014da8,
> inserted=0x7ffe8dc30548, phdr=0x7ffe8dc30550, propdb=0x100cfc0) at props.c:457
> #14 dav_get_props (propdb=0x100cfc0, doc=<optimized out>) at props.c:871
> #15 0x00007f8110cdf7c9 in dav_propfind_walker (wres=0x7ffe8dc307b8, calltype=<optimized out>) at mod_dav.c:2055
> #16 0x00007f8110be4885 in dav_fs_walker (fsctx=fsctx@entry=0x7ffe8dc307b0, depth=depth@entry=1) at repos.c:1600
> #17 0x00007f8110be4df9 in dav_fs_internal_walk (params=0x7ffe8dc30a50, depth=1, is_move=0, root_dst=0x0,
> response=0x7ffe8dc30a48) at repos.c:1844
> #18 0x00007f8110ce083b in dav_method_propfind (r=r@entry=0x100afb0) at mod_dav.c:2192
> #19 0x00007f8110ce37f8 in dav_handler (r=0x100afb0) at mod_dav.c:4814
>
>
Regards
Rüdiger
Re: svn commit: r1841225 -
/httpd/httpd/trunk/modules/dav/main/props.c
Posted by Joe Orton <jo...@redhat.com>.
On Fri, Feb 08, 2019 at 08:07:57AM +0100, Ruediger Pluem wrote:
> On 11/13/2018 10:26 AM, Joe Orton wrote:
> > On Mon, Nov 12, 2018 at 08:26:48AM +0100, Ruediger Pluem wrote:
> >> The discussion died a little bit, because of the other issue (frequent
> >> writeev calls). I know that the liveprops issue is not fixed yet, but
> >> I guess it makes sense if you commit the patch you posted here
> >> already.
> >
> > Sorry, I was looking at this again and the repro case I thought worked
> > was still showing leaks, so I got stuck. Will come back to it hopefully
> > later this week.
> >
>
> Going through the STATUS of 2.4.x I got aware that this died a little bit again.
> Any new ideas in the meantime?
I spent another half an afternoon looking at this.
I'm trying a PROPFIND/depth: 1 for *just* DAV: getcontenttype - and I
still see steadily increasing memory use during the response with trunk
or trunk plus my patch.
If I break in sbrk the memory allocation which triggers a heap expansion
is within some r->pool for the subrequest, so I suppose the bug is
around subreq handling somehow, but it's far from obvious.
#0 0x00007f8111b60130 in sbrk () from /lib64/libc.so.6
#1 0x00007f8111af54cd in __default_morecore () from /lib64/libc.so.6
#2 0x00007f8111af18a3 in sysmalloc () from /lib64/libc.so.6
#3 0x00007f8111af2a3f in _int_malloc () from /lib64/libc.so.6
#4 0x00007f8111af3b8f in malloc () from /lib64/libc.so.6
#5 0x00007f8111c7cd34 in allocator_alloc (in_size=8152, allocator=0xfd69f0) at memory/unix/apr_pools.c:411
#6 apr_pool_create_ex (newpool=newpool@entry=0x7ffe8dc303f8, parent=0x16fe6b8, abort_fn=0x440d90 <abort_on_oom>,
abort_fn@entry=0x0, allocator=0xfd69f0, allocator@entry=0x0) at memory/unix/apr_pools.c:1079
#7 0x0000000000461a4a in ap_location_walk (r=r@entry=0x16fe730) at request.c:1487
#8 0x0000000000462164 in ap_process_request_internal (r=0x16fe730) at request.c:238
#9 0x0000000000462cc8 in ap_sub_req_method_uri (method=method@entry=0x47a6a0 "GET",
new_uri=0x16fc6a8 "/modules/dav/file.b626.txt", r=0x100afb0, next_filter=next_filter@entry=0x0) at request.c:2346
#10 0x0000000000462d03 in ap_sub_req_lookup_uri (new_uri=<optimized out>, r=<optimized out>,
next_filter=next_filter@entry=0x0) at request.c:2359
#11 0x00007f8110ce6448 in dav_do_prop_subreq (propdb=0x100cfc0) at props.c:331
#12 0x00007f8110ce665d in dav_insert_coreprop (propdb=propdb@entry=0x100cfc0, propid=<optimized out>,
name=0x1014e18 "getcontenttype", what=what@entry=DAV_PROP_INSERT_VALUE, phdr=phdr@entry=0x7ffe8dc30550,
inserted=inserted@entry=0x7ffe8dc30548) at props.c:390
#13 0x00007f8110ce73b5 in dav_insert_liveprop (what=DAV_PROP_INSERT_VALUE, elem=0x1014da8, elem=0x1014da8,
inserted=0x7ffe8dc30548, phdr=0x7ffe8dc30550, propdb=0x100cfc0) at props.c:457
#14 dav_get_props (propdb=0x100cfc0, doc=<optimized out>) at props.c:871
#15 0x00007f8110cdf7c9 in dav_propfind_walker (wres=0x7ffe8dc307b8, calltype=<optimized out>) at mod_dav.c:2055
#16 0x00007f8110be4885 in dav_fs_walker (fsctx=fsctx@entry=0x7ffe8dc307b0, depth=depth@entry=1) at repos.c:1600
#17 0x00007f8110be4df9 in dav_fs_internal_walk (params=0x7ffe8dc30a50, depth=1, is_move=0, root_dst=0x0,
response=0x7ffe8dc30a48) at repos.c:1844
#18 0x00007f8110ce083b in dav_method_propfind (r=r@entry=0x100afb0) at mod_dav.c:2192
#19 0x00007f8110ce37f8 in dav_handler (r=0x100afb0) at mod_dav.c:4814