memory use of svn log -v with long changed paths lists

[Stefan Sperling <st...@elego.de>]

I'm still seeing 180MB of memory used by svn log -v with ruby's r11708
over ra_local with fsfs7-unpacked (120MB with fsfs6-unpacked).
The total output generated by svn log is only 5.5MB in size.
That seems somewhat disproportionate.

Of course, I'm happy that this invocation of svn log isn't running
out of memory anymore on my machine, as it did before r1604188!
So that's a huge step forward but I suppose we can still do better.

It seems caching is partly responsible for the memory growth.
I'm seeing huge jumps during svn_cache__set() calls, though we might
still be missing some scratch pools or iterpools to keep memory
usage low.

Does anyone know what can be done about this?

Re: memory use of svn log -v with long changed paths lists

[Stefan Fuhrmann <st...@wandisco.com>]

On Tue, Jun 24, 2014 at 11:04 AM, Stefan Sperling <st...@elego.de> wrote:

> On Tue, Jun 24, 2014 at 12:50:52AM +0200, Stefan Fuhrmann wrote:
> > The serializer itself uses a single, auto-growing buffer.
> > It should show up as a temporary peak, though.
>
> There is a peak roughly above 200MB before the log message
> is printed. It then settles at 173MB for me.
>
> > The current 100 bytes estimation / change is way to low
> > with the basic struct already taking 144 bytes on a 64bit
> > system. r1604947 should help here.
>
> It doesn't seem to help me, unfortunately.
> I'm still seeing 173MB of memory in top(1) while the changed
> paths list is printed and it goes down from there when the
> process exits. Do you see something else?
>

r1604947 only helps with the temp. peak during
serialization in a scratch pool.


> > > Does anyone know what can be done about this?
> > >
> >
> > Use a similar estimate as in r1604947 to skip svn_cache__set
> > for uncachable items (call svn_cache__is_cachable).
> >
> > As soon as I arrived in UK, I'll look into this.
>
> Thanks! I appreciate your help with this.
>

On my system, I'm now (r1605195) at 46MB peak,
55MB when I disable the cache bypass of r1605191.
Lower usage is possible but requires a completely
different, streaming, API.

-- Stefan^2.

Re: memory use of svn log -v with long changed paths lists

[Stefan Sperling <st...@elego.de>]

On Tue, Jun 24, 2014 at 12:50:52AM +0200, Stefan Fuhrmann wrote:
> The serializer itself uses a single, auto-growing buffer.
> It should show up as a temporary peak, though.

There is a peak roughly above 200MB before the log message
is printed. It then settles at 173MB for me.
 
> The current 100 bytes estimation / change is way to low
> with the basic struct already taking 144 bytes on a 64bit
> system. r1604947 should help here.

It doesn't seem to help me, unfortunately.
I'm still seeing 173MB of memory in top(1) while the changed
paths list is printed and it goes down from there when the
process exits. Do you see something else?

> > Does anyone know what can be done about this?
> >
> 
> Use a similar estimate as in r1604947 to skip svn_cache__set
> for uncachable items (call svn_cache__is_cachable).
> 
> As soon as I arrived in UK, I'll look into this.

Thanks! I appreciate your help with this.

Re: memory use of svn log -v with long changed paths lists

[Stefan Fuhrmann <st...@wandisco.com>]

On Mon, Jun 23, 2014 at 11:55 PM, Stefan Sperling <st...@elego.de> wrote:

> I'm still seeing 180MB of memory used by svn log -v with ruby's r11708
> over ra_local with fsfs7-unpacked (120MB with fsfs6-unpacked).
> The total output generated by svn log is only 5.5MB in size.
> That seems somewhat disproportionate.
>

Yes, it is. The id_t struct is somewhat heavy but in total
a change should be less than 200 bytes. So, everything
that needs more than ~30MB for the ruby example is
suspicious.


> Of course, I'm happy that this invocation of svn log isn't running
> out of memory anymore on my machine, as it did before r1604188!
> So that's a huge step forward but I suppose we can still do better.
>
> It seems caching is partly responsible for the memory growth.
> I'm seeing huge jumps during svn_cache__set() calls, though we might
> still be missing some scratch pools or iterpools to keep memory
> usage low.
>

The serializer itself uses a single, auto-growing buffer.
It should show up as a temporary peak, though.

The current 100 bytes estimation / change is way to low
with the basic struct already taking 144 bytes on a 64bit
system. r1604947 should help here.


> Does anyone know what can be done about this?
>

Use a similar estimate as in r1604947 to skip svn_cache__set
for uncachable items (call svn_cache__is_cachable).

As soon as I arrived in UK, I'll look into this. It's probably
at the point now where I need to counting bytes at every
step to spot inappropriate memory usage.

-- Stefan^2.