You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Vincenzo D'Amore <v....@gmail.com> on 2023/03/08 11:16:26 UTC
Solr 4.10 - master/slave replication
Hi all,
I'm working on a Solr 4.10 master/slave cluster where every night around
100 cores are indexed and replicated. This is the configuration:
master -> repeater -> slaves (8)
During the last 6 months the number of cores to be replicated almost
doubled and the slaves became very slow due very long garbage collection.
This situation is leading to having unresponsive slaves that return to work
normally only after a reboot.
In your experience do you know how I could limit the number of cores to be
replicated?
If there is some kind of configuration in the slaves that limits the number
of concurrent replications that can be activated.
Best regards,
Vincenzo
--
Vincenzo D'Amore
Re: Solr 4.10 - master/slave replication
Posted by David Hastings <ha...@gmail.com>.
Also take a look at your Xmx and Xms values. I find this to be absolutely
vital when taking into consideration the machine's own memory, and the
index size. you want the OS to be able to put most of the index segments
into memory, and want those two values (Xmx and Xms) to be identical,
Just a thought.
-Dave
On Wed, Mar 8, 2023 at 10:19 AM Vincenzo D'Amore <v....@gmail.com> wrote:
> Hi Shawn, thanks for the answer. Unfortunately the upgrade is not a viable
> option and the cluster is using an old JVM 1.7.
> On the other hand, as you suggested, we are changing the update procedure
> to not update all the indexes in one shot and after every index build (or
> bunch of indexes) starting the propagation process.
> This I hope will relax the memory consumption and make the slave more
> reliable.
>
> On Wed, Mar 8, 2023 at 3:37 PM Shawn Heisey <ap...@elyograg.org> wrote:
>
> > On 3/8/23 04:16, Vincenzo D'Amore wrote:
> > > I'm working on a Solr 4.10 master/slave cluster where every night
> around
> > > 100 cores are indexed and replicated. This is the configuration:
> >
> > I hope you're aware that 4.10.0 was released 8.5 years ago. It's been
> > end of life for a very long time. Is upgrading to 9.x a viable option?
> >
> > > During the last 6 months the number of cores to be replicated almost
> > > doubled and the slaves became very slow due very long garbage
> collection.
> >
> > What version of Java are you using to run that? Have you tuned the
> > garbage collection? Solr 4.10 does have GC tuning out of the box, but
> > I've never seen GC options like that before, and I have no idea whether
> > they are effective. If you're not running on Java 11 or later, I would
> > strongly recommend G1GC. You would put this in solr.in.sh:
> >
> > GC_TUNE=" \
> > -XX:+UseG1GC \
> > -XX:+ParallelRefProcEnabled \
> > -XX:G1HeapRegionSize=16m \
> > -XX:MaxGCPauseMillis=100 \
> > -XX:+ExplicitGCInvokesConcurrent \
> > -XX:+UseStringDeduplication \
> > -XX:+AlwaysPreTouch \
> > -XX:+UseNUMA \
> > "
> >
> > If you ARE running Java 11 or later, I would go with ZGC. I personally
> > wouldn't run a Solr version that old with anything newer than Java 8,
> > because Solr 7.0 was the first release that was guaranteed compatible
> > with Java 9.
> >
> > GC_TUNE=" \
> > -XX:+UnlockExperimentalVMOptions \
> > -XX:+UseZGC \
> > -XX:+ParallelRefProcEnabled \
> > -XX:+ExplicitGCInvokesConcurrent \
> > -XX:+UseStringDeduplication \
> > -XX:+AlwaysPreTouch \
> > -XX:+UseNUMA \
> > "
> >
> > > This situation is leading to having unresponsive slaves that return to
> > work
> > > normally only after a reboot.
> > > In your experience do you know how I could limit the number of cores to
> > be
> > > replicated?
> >
> > I haven't ever heard of something like that. Would be nice to have, but
> > if we did implement it, it would be in a 9.x version only, not applied
> > to anything older.
> >
> >
> > You should consider changing your nightly update procedure to not update
> > all the indexes at once. Tuning the GC would probably also help. The
> > options I have shown above are very good at eliminating most full GCs.
> > You should also consider increasing the heap size so there's more memory
> > for Java to work with. Can't make a recommendation about the heap size
> > without more info.
> >
> > Thanks,
> > Shawn
> >
>
>
> --
> Vincenzo D'Amore
>
Re: Solr 4.10 - master/slave replication
Posted by Vincenzo D'Amore <v....@gmail.com>.
Hi Shawn, thanks for the answer. Unfortunately the upgrade is not a viable
option and the cluster is using an old JVM 1.7.
On the other hand, as you suggested, we are changing the update procedure
to not update all the indexes in one shot and after every index build (or
bunch of indexes) starting the propagation process.
This I hope will relax the memory consumption and make the slave more
reliable.
On Wed, Mar 8, 2023 at 3:37 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 3/8/23 04:16, Vincenzo D'Amore wrote:
> > I'm working on a Solr 4.10 master/slave cluster where every night around
> > 100 cores are indexed and replicated. This is the configuration:
>
> I hope you're aware that 4.10.0 was released 8.5 years ago. It's been
> end of life for a very long time. Is upgrading to 9.x a viable option?
>
> > During the last 6 months the number of cores to be replicated almost
> > doubled and the slaves became very slow due very long garbage collection.
>
> What version of Java are you using to run that? Have you tuned the
> garbage collection? Solr 4.10 does have GC tuning out of the box, but
> I've never seen GC options like that before, and I have no idea whether
> they are effective. If you're not running on Java 11 or later, I would
> strongly recommend G1GC. You would put this in solr.in.sh:
>
> GC_TUNE=" \
> -XX:+UseG1GC \
> -XX:+ParallelRefProcEnabled \
> -XX:G1HeapRegionSize=16m \
> -XX:MaxGCPauseMillis=100 \
> -XX:+ExplicitGCInvokesConcurrent \
> -XX:+UseStringDeduplication \
> -XX:+AlwaysPreTouch \
> -XX:+UseNUMA \
> "
>
> If you ARE running Java 11 or later, I would go with ZGC. I personally
> wouldn't run a Solr version that old with anything newer than Java 8,
> because Solr 7.0 was the first release that was guaranteed compatible
> with Java 9.
>
> GC_TUNE=" \
> -XX:+UnlockExperimentalVMOptions \
> -XX:+UseZGC \
> -XX:+ParallelRefProcEnabled \
> -XX:+ExplicitGCInvokesConcurrent \
> -XX:+UseStringDeduplication \
> -XX:+AlwaysPreTouch \
> -XX:+UseNUMA \
> "
>
> > This situation is leading to having unresponsive slaves that return to
> work
> > normally only after a reboot.
> > In your experience do you know how I could limit the number of cores to
> be
> > replicated?
>
> I haven't ever heard of something like that. Would be nice to have, but
> if we did implement it, it would be in a 9.x version only, not applied
> to anything older.
>
>
> You should consider changing your nightly update procedure to not update
> all the indexes at once. Tuning the GC would probably also help. The
> options I have shown above are very good at eliminating most full GCs.
> You should also consider increasing the heap size so there's more memory
> for Java to work with. Can't make a recommendation about the heap size
> without more info.
>
> Thanks,
> Shawn
>
--
Vincenzo D'Amore
Re: Solr 4.10 - master/slave replication
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/8/23 04:16, Vincenzo D'Amore wrote:
> I'm working on a Solr 4.10 master/slave cluster where every night around
> 100 cores are indexed and replicated. This is the configuration:
I hope you're aware that 4.10.0 was released 8.5 years ago. It's been
end of life for a very long time. Is upgrading to 9.x a viable option?
> During the last 6 months the number of cores to be replicated almost
> doubled and the slaves became very slow due very long garbage collection.
What version of Java are you using to run that? Have you tuned the
garbage collection? Solr 4.10 does have GC tuning out of the box, but
I've never seen GC options like that before, and I have no idea whether
they are effective. If you're not running on Java 11 or later, I would
strongly recommend G1GC. You would put this in solr.in.sh:
GC_TUNE=" \
-XX:+UseG1GC \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=16m \
-XX:MaxGCPauseMillis=100 \
-XX:+ExplicitGCInvokesConcurrent \
-XX:+UseStringDeduplication \
-XX:+AlwaysPreTouch \
-XX:+UseNUMA \
"
If you ARE running Java 11 or later, I would go with ZGC. I personally
wouldn't run a Solr version that old with anything newer than Java 8,
because Solr 7.0 was the first release that was guaranteed compatible
with Java 9.
GC_TUNE=" \
-XX:+UnlockExperimentalVMOptions \
-XX:+UseZGC \
-XX:+ParallelRefProcEnabled \
-XX:+ExplicitGCInvokesConcurrent \
-XX:+UseStringDeduplication \
-XX:+AlwaysPreTouch \
-XX:+UseNUMA \
"
> This situation is leading to having unresponsive slaves that return to work
> normally only after a reboot.
> In your experience do you know how I could limit the number of cores to be
> replicated?
I haven't ever heard of something like that. Would be nice to have, but
if we did implement it, it would be in a 9.x version only, not applied
to anything older.
You should consider changing your nightly update procedure to not update
all the indexes at once. Tuning the GC would probably also help. The
options I have shown above are very good at eliminating most full GCs.
You should also consider increasing the heap size so there's more memory
for Java to work with. Can't make a recommendation about the heap size
without more info.
Thanks,
Shawn