You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Bram Van Dam <br...@intix.eu> on 2022/06/09 15:34:27 UTC
UpdateLog memory usage on startup
Howdy,
We've noticed that enabling larger transaction logs causes the memory
requirements for Solr to increase: Solr consumes large amounts of memory
at startup.
After procuring a heap dump, this seems to be because Solr initializes
ArrayLists in UpdateLog::getVersions with size
maxNbTransactionLogEntries. While this may be more efficient if your
actual log files are close to maximum size, this wastes memory when the
actual logs are small. This is something that occurs frequently when you
have a small number of shards which receive a lot of writes, and a lot
of shards which receive few (or no) writes.
We've seen cases where Solr needs an additional 10GiB of memory during
startup. It gets freed afterwards, but it does make startup painful.
The fix for SOLR-15676 further increased the memory footprint by
allocating a LongSet of the same size.
public List<Long> getVersions(int n, long maxVersion) {
List<Long> ret = new ArrayList<>(n);
LongSet set = new LongSet(n);
The naïve fix would be to simply replace this init of new ArrayList<>(n)
with new ArrayList<>(). ArrayList grows its capacity by 50% every time
it's full, resulting in some extra garbage overhead and extra calls to
array copy.
A quick bit of napkin math shows that for 10M entries, the array will
have to be reallocated 35 times. In our case, this is worth the extra
overhead. In the general case, it might not be?
Does anyone have any further insights?
Thanks,
- Bram
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: UpdateLog memory usage on startup
Posted by David Smiley <ds...@apache.org>.
The migration you speak of is in Lucene, not Solr. It would be noticed by
"Watchers".
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Tue, Jul 19, 2022 at 12:29 PM Ishan Chattopadhyaya <
ichattopadhyaya@gmail.com> wrote:
> Great find! Lets have it committed.
>
> On Tue, Jul 19, 2022 at 9:49 PM Bram Van Dam <br...@intix.eu> wrote:
>
> > Thanks for your reply, David. Given the apparent migration from from
> > Jira->Github, I didn't think that would get more response than the
> > mailing list 😅
> >
> > We've been running a patched version of 7.7 with a smaller Versions
> > arraylist for a while now, without any ill effects.
> >
> > - Bram
> >
> > On 13/07/2022 23.54, David Smiley wrote:
> > > Makes sense Bram.
> > >
> > > I note that it's been over a month with no response. Just a suggestion
> > --
> > > try commenting on the pertinent JIRA because it will get the attention
> of
> > > the last committer (and interested parties).
> > > BTW we could cap the initial ArrayList size to, say, Math.min(1024,n)
> > >
> > > ~ David Smiley
> > > Apache Lucene/Solr Search Developer
> > > http://www.linkedin.com/in/davidwsmiley
> > >
> > >
> > > On Thu, Jun 9, 2022 at 11:34 AM Bram Van Dam <br...@intix.eu>
> > wrote:
> > >
> > >> Howdy,
> > >>
> > >> We've noticed that enabling larger transaction logs causes the memory
> > >> requirements for Solr to increase: Solr consumes large amounts of
> memory
> > >> at startup.
> > >>
> > >> After procuring a heap dump, this seems to be because Solr initializes
> > >> ArrayLists in UpdateLog::getVersions with size
> > >> maxNbTransactionLogEntries. While this may be more efficient if your
> > >> actual log files are close to maximum size, this wastes memory when
> the
> > >> actual logs are small. This is something that occurs frequently when
> you
> > >> have a small number of shards which receive a lot of writes, and a lot
> > >> of shards which receive few (or no) writes.
> > >>
> > >> We've seen cases where Solr needs an additional 10GiB of memory during
> > >> startup. It gets freed afterwards, but it does make startup painful.
> > >>
> > >> The fix for SOLR-15676 further increased the memory footprint by
> > >> allocating a LongSet of the same size.
> > >>
> > >> public List<Long> getVersions(int n, long maxVersion) {
> > >> List<Long> ret = new ArrayList<>(n);
> > >> LongSet set = new LongSet(n);
> > >>
> > >> The naïve fix would be to simply replace this init of new
> ArrayList<>(n)
> > >> with new ArrayList<>(). ArrayList grows its capacity by 50% every time
> > >> it's full, resulting in some extra garbage overhead and extra calls to
> > >> array copy.
> > >>
> > >> A quick bit of napkin math shows that for 10M entries, the array will
> > >> have to be reallocated 35 times. In our case, this is worth the extra
> > >> overhead. In the general case, it might not be?
> > >>
> > >> Does anyone have any further insights?
> > >>
> > >> Thanks,
> > >>
> > >> - Bram
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > >> For additional commands, e-mail: dev-help@solr.apache.org
> > >>
> > >>
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
>
Re: UpdateLog memory usage on startup
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
Great find! Lets have it committed.
On Tue, Jul 19, 2022 at 9:49 PM Bram Van Dam <br...@intix.eu> wrote:
> Thanks for your reply, David. Given the apparent migration from from
> Jira->Github, I didn't think that would get more response than the
> mailing list 😅
>
> We've been running a patched version of 7.7 with a smaller Versions
> arraylist for a while now, without any ill effects.
>
> - Bram
>
> On 13/07/2022 23.54, David Smiley wrote:
> > Makes sense Bram.
> >
> > I note that it's been over a month with no response. Just a suggestion
> --
> > try commenting on the pertinent JIRA because it will get the attention of
> > the last committer (and interested parties).
> > BTW we could cap the initial ArrayList size to, say, Math.min(1024,n)
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
> >
> > On Thu, Jun 9, 2022 at 11:34 AM Bram Van Dam <br...@intix.eu>
> wrote:
> >
> >> Howdy,
> >>
> >> We've noticed that enabling larger transaction logs causes the memory
> >> requirements for Solr to increase: Solr consumes large amounts of memory
> >> at startup.
> >>
> >> After procuring a heap dump, this seems to be because Solr initializes
> >> ArrayLists in UpdateLog::getVersions with size
> >> maxNbTransactionLogEntries. While this may be more efficient if your
> >> actual log files are close to maximum size, this wastes memory when the
> >> actual logs are small. This is something that occurs frequently when you
> >> have a small number of shards which receive a lot of writes, and a lot
> >> of shards which receive few (or no) writes.
> >>
> >> We've seen cases where Solr needs an additional 10GiB of memory during
> >> startup. It gets freed afterwards, but it does make startup painful.
> >>
> >> The fix for SOLR-15676 further increased the memory footprint by
> >> allocating a LongSet of the same size.
> >>
> >> public List<Long> getVersions(int n, long maxVersion) {
> >> List<Long> ret = new ArrayList<>(n);
> >> LongSet set = new LongSet(n);
> >>
> >> The naïve fix would be to simply replace this init of new ArrayList<>(n)
> >> with new ArrayList<>(). ArrayList grows its capacity by 50% every time
> >> it's full, resulting in some extra garbage overhead and extra calls to
> >> array copy.
> >>
> >> A quick bit of napkin math shows that for 10M entries, the array will
> >> have to be reallocated 35 times. In our case, this is worth the extra
> >> overhead. In the general case, it might not be?
> >>
> >> Does anyone have any further insights?
> >>
> >> Thanks,
> >>
> >> - Bram
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> >> For additional commands, e-mail: dev-help@solr.apache.org
> >>
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: UpdateLog memory usage on startup
Posted by Bram Van Dam <br...@intix.eu>.
Thanks for your reply, David. Given the apparent migration from from
Jira->Github, I didn't think that would get more response than the
mailing list 😅
We've been running a patched version of 7.7 with a smaller Versions
arraylist for a while now, without any ill effects.
- Bram
On 13/07/2022 23.54, David Smiley wrote:
> Makes sense Bram.
>
> I note that it's been over a month with no response. Just a suggestion --
> try commenting on the pertinent JIRA because it will get the attention of
> the last committer (and interested parties).
> BTW we could cap the initial ArrayList size to, say, Math.min(1024,n)
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Thu, Jun 9, 2022 at 11:34 AM Bram Van Dam <br...@intix.eu> wrote:
>
>> Howdy,
>>
>> We've noticed that enabling larger transaction logs causes the memory
>> requirements for Solr to increase: Solr consumes large amounts of memory
>> at startup.
>>
>> After procuring a heap dump, this seems to be because Solr initializes
>> ArrayLists in UpdateLog::getVersions with size
>> maxNbTransactionLogEntries. While this may be more efficient if your
>> actual log files are close to maximum size, this wastes memory when the
>> actual logs are small. This is something that occurs frequently when you
>> have a small number of shards which receive a lot of writes, and a lot
>> of shards which receive few (or no) writes.
>>
>> We've seen cases where Solr needs an additional 10GiB of memory during
>> startup. It gets freed afterwards, but it does make startup painful.
>>
>> The fix for SOLR-15676 further increased the memory footprint by
>> allocating a LongSet of the same size.
>>
>> public List<Long> getVersions(int n, long maxVersion) {
>> List<Long> ret = new ArrayList<>(n);
>> LongSet set = new LongSet(n);
>>
>> The naïve fix would be to simply replace this init of new ArrayList<>(n)
>> with new ArrayList<>(). ArrayList grows its capacity by 50% every time
>> it's full, resulting in some extra garbage overhead and extra calls to
>> array copy.
>>
>> A quick bit of napkin math shows that for 10M entries, the array will
>> have to be reallocated 35 times. In our case, this is worth the extra
>> overhead. In the general case, it might not be?
>>
>> Does anyone have any further insights?
>>
>> Thanks,
>>
>> - Bram
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
>> For additional commands, e-mail: dev-help@solr.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: UpdateLog memory usage on startup
Posted by David Smiley <ds...@apache.org>.
Makes sense Bram.
I note that it's been over a month with no response. Just a suggestion --
try commenting on the pertinent JIRA because it will get the attention of
the last committer (and interested parties).
BTW we could cap the initial ArrayList size to, say, Math.min(1024,n)
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Thu, Jun 9, 2022 at 11:34 AM Bram Van Dam <br...@intix.eu> wrote:
> Howdy,
>
> We've noticed that enabling larger transaction logs causes the memory
> requirements for Solr to increase: Solr consumes large amounts of memory
> at startup.
>
> After procuring a heap dump, this seems to be because Solr initializes
> ArrayLists in UpdateLog::getVersions with size
> maxNbTransactionLogEntries. While this may be more efficient if your
> actual log files are close to maximum size, this wastes memory when the
> actual logs are small. This is something that occurs frequently when you
> have a small number of shards which receive a lot of writes, and a lot
> of shards which receive few (or no) writes.
>
> We've seen cases where Solr needs an additional 10GiB of memory during
> startup. It gets freed afterwards, but it does make startup painful.
>
> The fix for SOLR-15676 further increased the memory footprint by
> allocating a LongSet of the same size.
>
> public List<Long> getVersions(int n, long maxVersion) {
> List<Long> ret = new ArrayList<>(n);
> LongSet set = new LongSet(n);
>
> The naïve fix would be to simply replace this init of new ArrayList<>(n)
> with new ArrayList<>(). ArrayList grows its capacity by 50% every time
> it's full, resulting in some extra garbage overhead and extra calls to
> array copy.
>
> A quick bit of napkin math shows that for 10M entries, the array will
> have to be reallocated 35 times. In our case, this is worth the extra
> overhead. In the general case, it might not be?
>
> Does anyone have any further insights?
>
> Thanks,
>
> - Bram
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>