You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucy.apache.org by Peter Karman <pe...@peknet.com> on 2010/01/24 03:25:04 UTC
memory use
I have a collection of about 90k+ docs totaling about 2.2G.
When I index with KS 0.30_072 I see memory use climb to about 80M and hold
relatively steady.
With svn trunk, I see memory use climb steadily throughout the 15 minutes or so
it takes to index, reaching 600M+.
Initially I thought this was due to the recent changes in trunk to SortExternal
and so didn't think much of it since I knew that feature was currently receiving
some attention to reduce memory use. But I didn't want possible other leaks to
go unreported.
So I guess my question is: is that kind of memory use to be expected?
--
Peter Karman . http://peknet.com/ . peter@peknet.com
Re: memory use
Posted by Peter Karman <pe...@peknet.com>.
Marvin Humphrey wrote on 1/24/10 12:49 PM:
> As of r5749, test_valgrind is fixed and passes. I found a few small memory
> leaks here and there, but nothing that would cause a jump from 80 MB to 600
> MB. Does this problem also persist after a "./Build realclean"?
yes.
however, I realize I wasn't comparing apples with apples. I ran 0.30072 with
fewer sortable fields, since it doesn't support sortable FullTextType, and I ran
svn trunk with all FullTextType fields sortable. So the memory use could be due
to that.
I will rerun more comparable config and report back.
--
Peter Karman . http://peknet.com/ . peter@peknet.com
Re: memory use
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Sat, Jan 23, 2010 at 07:04:02PM -0800, Marvin Humphrey wrote:
> That's why we have test_valgrind... only the suppressions file
> trunk/devel/conf/kinoperl.supp has been wrong since the switch to
> LockFreeRegistry, so test_valgrind is currently broken.
As of r5749, test_valgrind is fixed and passes. I found a few small memory
leaks here and there, but nothing that would cause a jump from 80 MB to 600
MB. Does this problem also persist after a "./Build realclean"?
Marvin Humphrey
Re: memory use
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Sat, Jan 23, 2010 at 08:25:04PM -0600, Peter Karman wrote:
> I have a collection of about 90k+ docs totaling about 2.2G.
>
> When I index with KS 0.30_072 I see memory use climb to about 80M and hold
> relatively steady.
>
> With svn trunk, I see memory use climb steadily throughout the 15 minutes
> or so it takes to index, reaching 600M+.
We gots us a memory leak, then. Not surprising. It's hard to avoid making
refcounting errors while developing.
That's why we have test_valgrind... only the suppressions file
trunk/devel/conf/kinoperl.supp has been wrong since the switch to
LockFreeRegistry, so test_valgrind is currently broken.
> Initially I thought this was due to the recent changes in trunk to
> SortExternal and so didn't think much of it since I knew that feature was
> currently receiving some attention to reduce memory use. But I didn't want
> possible other leaks to go unreported.
>
> So I guess my question is: is that kind of memory use to be expected?
If you have sortable fields, then high memory usage is to be expected until we
fix SortWriter. However, SortWriter hasn't changed fundamentally in between
0.30_72 and now. So it would seem that this is a leak, almost certainly due
to a refcounting error.
Marvin Humphrey