You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Jesús Roca <xe...@gmail.com> on 2022/05/18 17:34:37 UTC

Growing cores after upgrade to 8.11.1

Hello,

We are having problema

We have a cluster with Solr 8 (15 nodes running RHEL) and ZooKeeper 3.6.2
(5 nodes) and only one collection of around 48 millions of docs with 10
shards and a replication factor of 3, so every server holds 2 cores.
A couple of weeks ago we performed an upgrade from Solr 8.9.0 to 8.11.1 and
we upgraded the Java version too from 1.8.0_322 to 11.0.14 and since then
some cores are constantly growing even when the number of docs is
practically the same.

I have upload an image where you can see how the size of a core grows over
time (we upgraded on 05/02), and another graphic where you can see the docs
number is quite stable compared to their size:
https://we.tl/t-pvV4RfXhYe
Currently the size of that replica of shard2 is 22,6 GB while leader shard
is 9,66 GB

We don't have any custom merge policy in solr.config file, we are using the
default merge policy.

When we restart an instance with a core that grows indefinitely it release
space. For ex, I have restarted the instance which graphic I shared and now
its size is 9,9 GB.
But after a while (some days) some cores grow again.

I have upload a COLSTATUS too with information about shard leaders and
their segments.

Any ideas about this issue?

Thanks!

Re: Growing cores after upgrade to 8.11.1

Posted by Jesús Roca <xe...@gmail.com>.
Sorry, it was my first time using WeTransfer.

Shared the files using google drive:

https://drive.google.com/drive/folders/1gndvdntxH1fzukUfA9H1PUIGidhT6KmU?usp=sharing

Thanks.

El mié, 18 may 2022 a las 19:55, Gus Heck (<gu...@gmail.com>) escribió:

> Your link leads to a signup page with advertising for clothing. Please
> don't do that.
>
> On Wed, May 18, 2022 at 1:35 PM Jesús Roca <xe...@gmail.com> wrote:
>
> > Hello,
> >
> > We are having problema
> >
> > We have a cluster with Solr 8 (15 nodes running RHEL) and ZooKeeper 3.6.2
> > (5 nodes) and only one collection of around 48 millions of docs with 10
> > shards and a replication factor of 3, so every server holds 2 cores.
> > A couple of weeks ago we performed an upgrade from Solr 8.9.0 to 8.11.1
> and
> > we upgraded the Java version too from 1.8.0_322 to 11.0.14 and since then
> > some cores are constantly growing even when the number of docs is
> > practically the same.
> >
> > I have upload an image where you can see how the size of a core grows
> over
> > time (we upgraded on 05/02), and another graphic where you can see the
> docs
> > number is quite stable compared to their size:
> > https://we.tl/t-pvV4RfXhYe
> > Currently the size of that replica of shard2 is 22,6 GB while leader
> shard
> > is 9,66 GB
> >
> > We don't have any custom merge policy in solr.config file, we are using
> the
> > default merge policy.
> >
> > When we restart an instance with a core that grows indefinitely it
> release
> > space. For ex, I have restarted the instance which graphic I shared and
> now
> > its size is 9,9 GB.
> > But after a while (some days) some cores grow again.
> >
> > I have upload a COLSTATUS too with information about shard leaders and
> > their segments.
> >
> > Any ideas about this issue?
> >
> > Thanks!
> >
>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>

Re: Growing cores after upgrade to 8.11.1

Posted by Gus Heck <gu...@gmail.com>.
Your link leads to a signup page with advertising for clothing. Please
don't do that.

On Wed, May 18, 2022 at 1:35 PM Jesús Roca <xe...@gmail.com> wrote:

> Hello,
>
> We are having problema
>
> We have a cluster with Solr 8 (15 nodes running RHEL) and ZooKeeper 3.6.2
> (5 nodes) and only one collection of around 48 millions of docs with 10
> shards and a replication factor of 3, so every server holds 2 cores.
> A couple of weeks ago we performed an upgrade from Solr 8.9.0 to 8.11.1 and
> we upgraded the Java version too from 1.8.0_322 to 11.0.14 and since then
> some cores are constantly growing even when the number of docs is
> practically the same.
>
> I have upload an image where you can see how the size of a core grows over
> time (we upgraded on 05/02), and another graphic where you can see the docs
> number is quite stable compared to their size:
> https://we.tl/t-pvV4RfXhYe
> Currently the size of that replica of shard2 is 22,6 GB while leader shard
> is 9,66 GB
>
> We don't have any custom merge policy in solr.config file, we are using the
> default merge policy.
>
> When we restart an instance with a core that grows indefinitely it release
> space. For ex, I have restarted the instance which graphic I shared and now
> its size is 9,9 GB.
> But after a while (some days) some cores grow again.
>
> I have upload a COLSTATUS too with information about shard leaders and
> their segments.
>
> Any ideas about this issue?
>
> Thanks!
>


-- 
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)

Re: Growing cores after upgrade to 8.11.1

Posted by Jesús Roca <xe...@gmail.com>.
Hi,

We have already performed a rolling restart of solr service in all nodes,
but not a rolling server restart.

Something that could be strange is that shard 7 is the only one with
minSegmentLuceneVersion to 8.9.0
All the others shards changed to 8.11.1 several days ago.

Regards.

El jue, 19 may 2022 a las 13:04, Ere Maijala (<er...@helsinki.fi>)
escribió:

> Hi,
>
> For whatever it's worth, I've occasionally seen this, or something
> similar, happening as well. Restarting Solr on the offending server
> always fixes the issue until it starts happening again. For some reason
> the problem seems to manifest after a rolling server restart (OS
> updates). I haven't been able to find anything interesting in Solr logs.
>
> --Ere
>
> Jesús Roca kirjoitti 18.5.2022 klo 20.34:
> > Hello,
> >
> > We are having problema
> >
> > We have a cluster with Solr 8 (15 nodes running RHEL) and ZooKeeper 3.6.2
> > (5 nodes) and only one collection of around 48 millions of docs with 10
> > shards and a replication factor of 3, so every server holds 2 cores.
> > A couple of weeks ago we performed an upgrade from Solr 8.9.0 to 8.11.1
> and
> > we upgraded the Java version too from 1.8.0_322 to 11.0.14 and since then
> > some cores are constantly growing even when the number of docs is
> > practically the same.
> >
> > I have upload an image where you can see how the size of a core grows
> over
> > time (we upgraded on 05/02), and another graphic where you can see the
> docs
> > number is quite stable compared to their size:
> > https://we.tl/t-pvV4RfXhYe
> > Currently the size of that replica of shard2 is 22,6 GB while leader
> shard
> > is 9,66 GB
> >
> > We don't have any custom merge policy in solr.config file, we are using
> the
> > default merge policy.
> >
> > When we restart an instance with a core that grows indefinitely it
> release
> > space. For ex, I have restarted the instance which graphic I shared and
> now
> > its size is 9,9 GB.
> > But after a while (some days) some cores grow again.
> >
> > I have upload a COLSTATUS too with information about shard leaders and
> > their segments.
> >
> > Any ideas about this issue?
> >
> > Thanks!
> >
>
> --
> Ere Maijala
> Kansalliskirjasto / The National Library of Finland
>

Re: Growing cores after upgrade to 8.11.1

Posted by Ere Maijala <er...@helsinki.fi>.
Hi,

For whatever it's worth, I've occasionally seen this, or something 
similar, happening as well. Restarting Solr on the offending server 
always fixes the issue until it starts happening again. For some reason 
the problem seems to manifest after a rolling server restart (OS 
updates). I haven't been able to find anything interesting in Solr logs.

--Ere

Jesús Roca kirjoitti 18.5.2022 klo 20.34:
> Hello,
> 
> We are having problema
> 
> We have a cluster with Solr 8 (15 nodes running RHEL) and ZooKeeper 3.6.2
> (5 nodes) and only one collection of around 48 millions of docs with 10
> shards and a replication factor of 3, so every server holds 2 cores.
> A couple of weeks ago we performed an upgrade from Solr 8.9.0 to 8.11.1 and
> we upgraded the Java version too from 1.8.0_322 to 11.0.14 and since then
> some cores are constantly growing even when the number of docs is
> practically the same.
> 
> I have upload an image where you can see how the size of a core grows over
> time (we upgraded on 05/02), and another graphic where you can see the docs
> number is quite stable compared to their size:
> https://we.tl/t-pvV4RfXhYe
> Currently the size of that replica of shard2 is 22,6 GB while leader shard
> is 9,66 GB
> 
> We don't have any custom merge policy in solr.config file, we are using the
> default merge policy.
> 
> When we restart an instance with a core that grows indefinitely it release
> space. For ex, I have restarted the instance which graphic I shared and now
> its size is 9,9 GB.
> But after a while (some days) some cores grow again.
> 
> I have upload a COLSTATUS too with information about shard leaders and
> their segments.
> 
> Any ideas about this issue?
> 
> Thanks!
> 

-- 
Ere Maijala
Kansalliskirjasto / The National Library of Finland