You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Satya Nand <sa...@indiamart.com.INVALID> on 2021/08/03 06:40:00 UTC
Big difference in response time on solr 8.7; Optimized vs un
Optimized core
Hi,
We have recently upgraded solr from version 6.5 to version 8.7. But
opposite to our expectations, the response time increased by 40 %.
On solr 8.7 the difference between optimized and unoptimized index is also
very huge. 350 ms on optimized and 650 ms on unoptimized. The difference is
only 5 GB in size in cores of optimized and unoptimized. The segment count
in the optimized index is 1 and 20 in the unoptimized index.
I wanted to ask, Is this normal behavior on solr 8.7, or was there some
setting that we forgot to add? Pleas also tell us how can we reduce the
response time in unoptimzed core.
*Specifications*
We are using master slave architecture, Polling interval is 3 hours
RAM- 96 GB
CPU-14
Heap-30 GB
Index Size-95 GB
Segments size-20
Merge Policy :
<mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
<int name="maxMergeAtOnce">5</int> <int name="segmentsPerTier">3</int> </
mergePolicyFactory>
--
Re: Big difference in response time on solr 8.7; Optimized vs un
Optimized core
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Hi Deepak,
Heap Size
index size(Almost Same)
schema
solr config
We have only done one change, Earlier we using synonym_edismax parser. As
this parser was not available for solr 8.7 we replaced it with edismax +
synonym graph filter factory to handle multiword synonym.
Also, On solr 8.7 the difference between optimized and unoptimized index is
also very huge. 180+ ms on optimized and 350+ ms on unoptimized. The
difference is only 5 GB in size in cores of optimized and unoptimized. The
segment count in the optimized index is 1 and 20 in the unoptimized index.
On Wed, Aug 11, 2021 at 12:39 PM Deepak Goel <de...@gmail.com> wrote:
> You will have to elaborate a bit on: "keeping everything same"
>
>
> Deepak
> "The greatness of a nation can be judged by the way its animals are treated
> - Mahatma Gandhi"
>
> +91 73500 12833
> deicool@gmail.com
>
> Facebook: https://www.facebook.com/deicool
> LinkedIn: www.linkedin.com/in/deicool
>
> "Plant a Tree, Go Green"
>
> Make In India : http://www.makeinindia.com/home
>
>
> On Wed, Aug 11, 2021 at 12:38 PM Satya Nand
> <sa...@indiamart.com.invalid> wrote:
>
> > Thanks, Deepak, We will try doing this.
> >
> > But still, I am wondering what led to the increase in response time this
> > much from solr 6.5 to solr 8.7, keeping everything same.
> > We are facing an increase of 100-150 ms.
> >
> >
> > On Wed, Aug 11, 2021 at 11:55 AM Deepak Goel <de...@gmail.com> wrote:
> >
> > > If I were you, then I would stick to the 128 GB machine. And then look
> at
> > > other parameters to tune...
> > >
> > >
> > > Deepak
> > > "The greatness of a nation can be judged by the way its animals are
> > treated
> > > - Mahatma Gandhi"
> > >
> > > +91 73500 12833
> > > deicool@gmail.com
> > >
> > > Facebook: https://www.facebook.com/deicool
> > > LinkedIn: www.linkedin.com/in/deicool
> > >
> > > "Plant a Tree, Go Green"
> > >
> > > Make In India : http://www.makeinindia.com/home
> > >
> > >
> > > On Tue, Aug 3, 2021 at 3:25 PM Satya Nand <satya.nand@indiamart.com
> > > .invalid>
> > > wrote:
> > >
> > > > Hi Deepak,
> > > >
> > > > We actually tried with 128 GB machine, Didn't help in response time.
> So
> > > we
> > > > moved back to 96GB.
> > > >
> > > > On Tue, Aug 3, 2021 at 2:11 PM Deepak Goel <de...@gmail.com>
> wrote:
> > > >
> > > > > I am confused a bit about the maths:
> > > > >
> > > > > Heap-30 GB & Index Size-95 GB is equal to 125GB. And the RAM is
> 96GB.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Deepak
> > > > > "The greatness of a nation can be judged by the way its animals are
> > > > treated
> > > > > - Mahatma Gandhi"
> > > > >
> > > > > +91 73500 12833
> > > > > deicool@gmail.com
> > > > >
> > > > > Facebook: https://www.facebook.com/deicool
> > > > > LinkedIn: www.linkedin.com/in/deicool
> > > > >
> > > > > "Plant a Tree, Go Green"
> > > > >
> > > > > Make In India : http://www.makeinindia.com/home
> > > > >
> > > > >
> > > > > On Tue, Aug 3, 2021 at 12:10 PM Satya Nand <
> satya.nand@indiamart.com
> > > > > .invalid>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We have recently upgraded solr from version 6.5 to version 8.7.
> > But
> > > > > > opposite to our expectations, the response time increased by 40
> %.
> > > > > >
> > > > > > On solr 8.7 the difference between optimized and unoptimized
> index
> > is
> > > > > also
> > > > > > very huge. 350 ms on optimized and 650 ms on unoptimized. The
> > > > difference
> > > > > is
> > > > > > only 5 GB in size in cores of optimized and unoptimized. The
> > segment
> > > > > count
> > > > > > in the optimized index is 1 and 20 in the unoptimized index.
> > > > > >
> > > > > > I wanted to ask, Is this normal behavior on solr 8.7, or was
> there
> > > some
> > > > > > setting that we forgot to add? Pleas also tell us how can we
> reduce
> > > the
> > > > > > response time in unoptimzed core.
> > > > > >
> > > > > > *Specifications*
> > > > > > We are using master slave architecture, Polling interval is 3
> hours
> > > > > > RAM- 96 GB
> > > > > > CPU-14
> > > > > > Heap-30 GB
> > > > > > Index Size-95 GB
> > > > > > Segments size-20
> > > > > > Merge Policy :
> > > > > >
> > > > > > <mergePolicyFactory
> > > > > class="org.apache.solr.index.TieredMergePolicyFactory">
> > > > > > <int name="maxMergeAtOnce">5</int> <int
> > > name="segmentsPerTier">3</int>
> > > > </
> > > > > > mergePolicyFactory>
> > > > > >
> > > > > > --
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > >
> > >
> >
> > --
> >
> >
>
--
Re: Big difference in response time on solr 8.7; Optimized vs un
Optimized core
Posted by Deepak Goel <de...@gmail.com>.
You will have to elaborate a bit on: "keeping everything same"
Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
Make In India : http://www.makeinindia.com/home
On Wed, Aug 11, 2021 at 12:38 PM Satya Nand
<sa...@indiamart.com.invalid> wrote:
> Thanks, Deepak, We will try doing this.
>
> But still, I am wondering what led to the increase in response time this
> much from solr 6.5 to solr 8.7, keeping everything same.
> We are facing an increase of 100-150 ms.
>
>
> On Wed, Aug 11, 2021 at 11:55 AM Deepak Goel <de...@gmail.com> wrote:
>
> > If I were you, then I would stick to the 128 GB machine. And then look at
> > other parameters to tune...
> >
> >
> > Deepak
> > "The greatness of a nation can be judged by the way its animals are
> treated
> > - Mahatma Gandhi"
> >
> > +91 73500 12833
> > deicool@gmail.com
> >
> > Facebook: https://www.facebook.com/deicool
> > LinkedIn: www.linkedin.com/in/deicool
> >
> > "Plant a Tree, Go Green"
> >
> > Make In India : http://www.makeinindia.com/home
> >
> >
> > On Tue, Aug 3, 2021 at 3:25 PM Satya Nand <satya.nand@indiamart.com
> > .invalid>
> > wrote:
> >
> > > Hi Deepak,
> > >
> > > We actually tried with 128 GB machine, Didn't help in response time. So
> > we
> > > moved back to 96GB.
> > >
> > > On Tue, Aug 3, 2021 at 2:11 PM Deepak Goel <de...@gmail.com> wrote:
> > >
> > > > I am confused a bit about the maths:
> > > >
> > > > Heap-30 GB & Index Size-95 GB is equal to 125GB. And the RAM is 96GB.
> > > >
> > > >
> > > >
> > > >
> > > > Deepak
> > > > "The greatness of a nation can be judged by the way its animals are
> > > treated
> > > > - Mahatma Gandhi"
> > > >
> > > > +91 73500 12833
> > > > deicool@gmail.com
> > > >
> > > > Facebook: https://www.facebook.com/deicool
> > > > LinkedIn: www.linkedin.com/in/deicool
> > > >
> > > > "Plant a Tree, Go Green"
> > > >
> > > > Make In India : http://www.makeinindia.com/home
> > > >
> > > >
> > > > On Tue, Aug 3, 2021 at 12:10 PM Satya Nand <satya.nand@indiamart.com
> > > > .invalid>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We have recently upgraded solr from version 6.5 to version 8.7.
> But
> > > > > opposite to our expectations, the response time increased by 40 %.
> > > > >
> > > > > On solr 8.7 the difference between optimized and unoptimized index
> is
> > > > also
> > > > > very huge. 350 ms on optimized and 650 ms on unoptimized. The
> > > difference
> > > > is
> > > > > only 5 GB in size in cores of optimized and unoptimized. The
> segment
> > > > count
> > > > > in the optimized index is 1 and 20 in the unoptimized index.
> > > > >
> > > > > I wanted to ask, Is this normal behavior on solr 8.7, or was there
> > some
> > > > > setting that we forgot to add? Pleas also tell us how can we reduce
> > the
> > > > > response time in unoptimzed core.
> > > > >
> > > > > *Specifications*
> > > > > We are using master slave architecture, Polling interval is 3 hours
> > > > > RAM- 96 GB
> > > > > CPU-14
> > > > > Heap-30 GB
> > > > > Index Size-95 GB
> > > > > Segments size-20
> > > > > Merge Policy :
> > > > >
> > > > > <mergePolicyFactory
> > > > class="org.apache.solr.index.TieredMergePolicyFactory">
> > > > > <int name="maxMergeAtOnce">5</int> <int
> > name="segmentsPerTier">3</int>
> > > </
> > > > > mergePolicyFactory>
> > > > >
> > > > > --
> > > > >
> > > > >
> > > >
> > >
> > > --
> > >
> > >
> >
>
> --
>
>
Re: Big difference in response time on solr 8.7; Optimized vs un
Optimized core
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Thanks, Deepak, We will try doing this.
But still, I am wondering what led to the increase in response time this
much from solr 6.5 to solr 8.7, keeping everything same.
We are facing an increase of 100-150 ms.
On Wed, Aug 11, 2021 at 11:55 AM Deepak Goel <de...@gmail.com> wrote:
> If I were you, then I would stick to the 128 GB machine. And then look at
> other parameters to tune...
>
>
> Deepak
> "The greatness of a nation can be judged by the way its animals are treated
> - Mahatma Gandhi"
>
> +91 73500 12833
> deicool@gmail.com
>
> Facebook: https://www.facebook.com/deicool
> LinkedIn: www.linkedin.com/in/deicool
>
> "Plant a Tree, Go Green"
>
> Make In India : http://www.makeinindia.com/home
>
>
> On Tue, Aug 3, 2021 at 3:25 PM Satya Nand <satya.nand@indiamart.com
> .invalid>
> wrote:
>
> > Hi Deepak,
> >
> > We actually tried with 128 GB machine, Didn't help in response time. So
> we
> > moved back to 96GB.
> >
> > On Tue, Aug 3, 2021 at 2:11 PM Deepak Goel <de...@gmail.com> wrote:
> >
> > > I am confused a bit about the maths:
> > >
> > > Heap-30 GB & Index Size-95 GB is equal to 125GB. And the RAM is 96GB.
> > >
> > >
> > >
> > >
> > > Deepak
> > > "The greatness of a nation can be judged by the way its animals are
> > treated
> > > - Mahatma Gandhi"
> > >
> > > +91 73500 12833
> > > deicool@gmail.com
> > >
> > > Facebook: https://www.facebook.com/deicool
> > > LinkedIn: www.linkedin.com/in/deicool
> > >
> > > "Plant a Tree, Go Green"
> > >
> > > Make In India : http://www.makeinindia.com/home
> > >
> > >
> > > On Tue, Aug 3, 2021 at 12:10 PM Satya Nand <satya.nand@indiamart.com
> > > .invalid>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > We have recently upgraded solr from version 6.5 to version 8.7. But
> > > > opposite to our expectations, the response time increased by 40 %.
> > > >
> > > > On solr 8.7 the difference between optimized and unoptimized index is
> > > also
> > > > very huge. 350 ms on optimized and 650 ms on unoptimized. The
> > difference
> > > is
> > > > only 5 GB in size in cores of optimized and unoptimized. The segment
> > > count
> > > > in the optimized index is 1 and 20 in the unoptimized index.
> > > >
> > > > I wanted to ask, Is this normal behavior on solr 8.7, or was there
> some
> > > > setting that we forgot to add? Pleas also tell us how can we reduce
> the
> > > > response time in unoptimzed core.
> > > >
> > > > *Specifications*
> > > > We are using master slave architecture, Polling interval is 3 hours
> > > > RAM- 96 GB
> > > > CPU-14
> > > > Heap-30 GB
> > > > Index Size-95 GB
> > > > Segments size-20
> > > > Merge Policy :
> > > >
> > > > <mergePolicyFactory
> > > class="org.apache.solr.index.TieredMergePolicyFactory">
> > > > <int name="maxMergeAtOnce">5</int> <int
> name="segmentsPerTier">3</int>
> > </
> > > > mergePolicyFactory>
> > > >
> > > > --
> > > >
> > > >
> > >
> >
> > --
> >
> >
>
--
Re: Big difference in response time on solr 8.7; Optimized vs un
Optimized core
Posted by Deepak Goel <de...@gmail.com>.
If I were you, then I would stick to the 128 GB machine. And then look at
other parameters to tune...
Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
Make In India : http://www.makeinindia.com/home
On Tue, Aug 3, 2021 at 3:25 PM Satya Nand <sa...@indiamart.com.invalid>
wrote:
> Hi Deepak,
>
> We actually tried with 128 GB machine, Didn't help in response time. So we
> moved back to 96GB.
>
> On Tue, Aug 3, 2021 at 2:11 PM Deepak Goel <de...@gmail.com> wrote:
>
> > I am confused a bit about the maths:
> >
> > Heap-30 GB & Index Size-95 GB is equal to 125GB. And the RAM is 96GB.
> >
> >
> >
> >
> > Deepak
> > "The greatness of a nation can be judged by the way its animals are
> treated
> > - Mahatma Gandhi"
> >
> > +91 73500 12833
> > deicool@gmail.com
> >
> > Facebook: https://www.facebook.com/deicool
> > LinkedIn: www.linkedin.com/in/deicool
> >
> > "Plant a Tree, Go Green"
> >
> > Make In India : http://www.makeinindia.com/home
> >
> >
> > On Tue, Aug 3, 2021 at 12:10 PM Satya Nand <satya.nand@indiamart.com
> > .invalid>
> > wrote:
> >
> > > Hi,
> > >
> > > We have recently upgraded solr from version 6.5 to version 8.7. But
> > > opposite to our expectations, the response time increased by 40 %.
> > >
> > > On solr 8.7 the difference between optimized and unoptimized index is
> > also
> > > very huge. 350 ms on optimized and 650 ms on unoptimized. The
> difference
> > is
> > > only 5 GB in size in cores of optimized and unoptimized. The segment
> > count
> > > in the optimized index is 1 and 20 in the unoptimized index.
> > >
> > > I wanted to ask, Is this normal behavior on solr 8.7, or was there some
> > > setting that we forgot to add? Pleas also tell us how can we reduce the
> > > response time in unoptimzed core.
> > >
> > > *Specifications*
> > > We are using master slave architecture, Polling interval is 3 hours
> > > RAM- 96 GB
> > > CPU-14
> > > Heap-30 GB
> > > Index Size-95 GB
> > > Segments size-20
> > > Merge Policy :
> > >
> > > <mergePolicyFactory
> > class="org.apache.solr.index.TieredMergePolicyFactory">
> > > <int name="maxMergeAtOnce">5</int> <int name="segmentsPerTier">3</int>
> </
> > > mergePolicyFactory>
> > >
> > > --
> > >
> > >
> >
>
> --
>
>
Re: Big difference in response time on solr 8.7; Optimized vs un
Optimized core
Posted by Satya Nand <sa...@indiamart.com.INVALID>.
Hi Deepak,
We actually tried with 128 GB machine, Didn't help in response time. So we
moved back to 96GB.
On Tue, Aug 3, 2021 at 2:11 PM Deepak Goel <de...@gmail.com> wrote:
> I am confused a bit about the maths:
>
> Heap-30 GB & Index Size-95 GB is equal to 125GB. And the RAM is 96GB.
>
>
>
>
> Deepak
> "The greatness of a nation can be judged by the way its animals are treated
> - Mahatma Gandhi"
>
> +91 73500 12833
> deicool@gmail.com
>
> Facebook: https://www.facebook.com/deicool
> LinkedIn: www.linkedin.com/in/deicool
>
> "Plant a Tree, Go Green"
>
> Make In India : http://www.makeinindia.com/home
>
>
> On Tue, Aug 3, 2021 at 12:10 PM Satya Nand <satya.nand@indiamart.com
> .invalid>
> wrote:
>
> > Hi,
> >
> > We have recently upgraded solr from version 6.5 to version 8.7. But
> > opposite to our expectations, the response time increased by 40 %.
> >
> > On solr 8.7 the difference between optimized and unoptimized index is
> also
> > very huge. 350 ms on optimized and 650 ms on unoptimized. The difference
> is
> > only 5 GB in size in cores of optimized and unoptimized. The segment
> count
> > in the optimized index is 1 and 20 in the unoptimized index.
> >
> > I wanted to ask, Is this normal behavior on solr 8.7, or was there some
> > setting that we forgot to add? Pleas also tell us how can we reduce the
> > response time in unoptimzed core.
> >
> > *Specifications*
> > We are using master slave architecture, Polling interval is 3 hours
> > RAM- 96 GB
> > CPU-14
> > Heap-30 GB
> > Index Size-95 GB
> > Segments size-20
> > Merge Policy :
> >
> > <mergePolicyFactory
> class="org.apache.solr.index.TieredMergePolicyFactory">
> > <int name="maxMergeAtOnce">5</int> <int name="segmentsPerTier">3</int> </
> > mergePolicyFactory>
> >
> > --
> >
> >
>
--
Re: Big difference in response time on solr 8.7; Optimized vs un
Optimized core
Posted by Deepak Goel <de...@gmail.com>.
I am confused a bit about the maths:
Heap-30 GB & Index Size-95 GB is equal to 125GB. And the RAM is 96GB.
Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
Make In India : http://www.makeinindia.com/home
On Tue, Aug 3, 2021 at 12:10 PM Satya Nand <sa...@indiamart.com.invalid>
wrote:
> Hi,
>
> We have recently upgraded solr from version 6.5 to version 8.7. But
> opposite to our expectations, the response time increased by 40 %.
>
> On solr 8.7 the difference between optimized and unoptimized index is also
> very huge. 350 ms on optimized and 650 ms on unoptimized. The difference is
> only 5 GB in size in cores of optimized and unoptimized. The segment count
> in the optimized index is 1 and 20 in the unoptimized index.
>
> I wanted to ask, Is this normal behavior on solr 8.7, or was there some
> setting that we forgot to add? Pleas also tell us how can we reduce the
> response time in unoptimzed core.
>
> *Specifications*
> We are using master slave architecture, Polling interval is 3 hours
> RAM- 96 GB
> CPU-14
> Heap-30 GB
> Index Size-95 GB
> Segments size-20
> Merge Policy :
>
> <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
> <int name="maxMergeAtOnce">5</int> <int name="segmentsPerTier">3</int> </
> mergePolicyFactory>
>
> --
>
>