You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Pradeep <ku...@gmail.com> on 2022/12/12 15:14:28 UTC
Slowness in Solr Optimize
Hi Team,
We have migrated Apache solr aws load balancer from classic to Application
load balancer. We havent changed anything in solr other than migrating load
balancer but solr taking more time while committing in Solr.
Fyi, we are processing 200 records every batch, this is failing at last
stage of commit in solr (deleting stale records in solr). But same logic
working fine from many years in classic load balancer but after migration
to ALB this is happenning.
Regards,
Pradeep
Re: Slowness in Solr Optimize
Posted by Dave <ha...@gmail.com>.
You can check the atones to see if solr itself actually slowed down. As solr has nothing to do with a load balancer I doubt it has. Also you used a sentence that concerns me, clearing out the deleted documents, which sounds like an optimize command. You as a user should never use that, let solr manage its index on its own and just make sure you always have three times the disk space as a full index
> On Dec 12, 2022, at 3:16 PM, Pradeep <ku...@gmail.com> wrote:
>
> Thank you shawn.
> I will have look, I have checked the solr.log file but didn't see any
> error, after load balancer changes we are seeing this issue.
>
> How to check time taken by solr to execute api? Also can you share me solr
> doc how we can query manually solr index for specific record or any api's
> you can provide.
>
>
> Thanks
>
>
>
>> On Mon, 12 Dec, 2022, 18:51 Shawn Heisey, <el...@elyograg.org> wrote:
>>
>>> On 12/12/22 08:14, Pradeep wrote:
>>> We have migrated Apache solr aws load balancer from classic to
>> Application
>>> load balancer. We havent changed anything in solr other than migrating
>> load
>>> balancer but solr taking more time while committing in Solr.
>>>
>>> Fyi, we are processing 200 records every batch, this is failing at last
>>> stage of commit in solr (deleting stale records in solr). But same logic
>>> working fine from many years in classic load balancer but after migration
>>> to ALB this is happenning.
>>
>> The message subject mentions optimize but the message body does not.
>>
>> Is optimize involved? An optimize of a large index is always going to
>> be slow.
>>
>> Solr does not have any kind of "deleting stale records" that I have ever
>> heard of, so that must be something your code, which is more difficult
>> for us to troubleshoot because we are not going to have any idea how to
>> get that code to give useful logs without a lot more information about
>> that code.
>>
>> If you're encountering an error from Solr then solr.log should have at
>> least some info about what happened and that might be enough info to
>> figure out the problem.
>>
>> If changing the load balancer caused issues, then you may need to talk
>> to someone who understands how to troubleshoot the LB.
>>
>> Basically we need a lot more information about what's happening to have
>> any hope of helping you.
>>
>> Thanks,
>> Shawn
>>
>>
Re: Slowness in Solr Optimize
Posted by David Hastings <ha...@gmail.com>.
Interesting, actually one method I used was to use Memcached to get the
next page of results stored in the web servers memory, worked pretty slick
if you had enough memory
On Tue, Dec 13, 2022 at 3:01 PM Walter Underwood <wu...@wunderwood.org>
wrote:
> Sticky sessions only help for later pages. They don’t direct new queries
> to servers that already have that query cached. That is a bigger win than
> second pages.
>
> I’ve thought about making a hash of the query-meaningful params (q, bq, …)
> and routing based on that, but that has other problems. When one server
> goes down, you simultaneously blow out every cache, which can cause
> cascading server overload. This is an actual problem, that took down the
> Infoseek search engine years ago. All the servers went down like dominos.
>
> There are some hashing approaches to fix this, but testing gets difficult.
>
> If you want really effective caching, put an HTTP cache in front of the
> load balancer. That is way better than fussing with request routing and the
> testing is minimal, because cache servers are designed to do exactly that.
> I did some benchmarks, and Varnish caching was slightly faster than the
> caching in a single Solr server.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
> > On Dec 13, 2022, at 11:27 AM, David Hastings <
> hastings.recursive@gmail.com> wrote:
> >
> > Ah, that makes sense. If you can do sticky sessions and such with your
> > balancers, plus I never had to deal with the throughput of something like
> > Netflix, so for mine and most use cases, I still feel one very hot server
> > is better than N warm ones.
> >
> > "but AWS load balancers aren’t very smart." - agreed, but it looks like
> > they attempted something for it:
> >
> https://docs.aws.amazon.com/elasticloadbalancing/latest/application/sticky-sessions.html
> >
> > but can you sticky session an app server to a solr server?
> > Like
> > user->loadbalancer->app server X->solr load blancer->solr server X
> > and make sure that app server X- and solr server X stay connected for the
> > user's session? would love to see how to configure this
> >
> >
> > On Tue, Dec 13, 2022 at 1:53 PM Walter Underwood <wu...@wunderwood.org>
> > wrote:
> >
> >> I’ve always run load balancers, starting with Solr 1.2 at Netflix.
> >> Failover (cold) spares have cold caches, so have slow performance until
> the
> >> cache fills. I configure N+1 capacity, where N servers can handle the
> >> expected load, then we add one for failure handling. All the spares are
> hot.
> >>
> >> I even run updates to Solr Cloud through a load balancer. It is easy to
> >> configure and Solr is very efficient at forwarding documents to shard
> >> leaders. It is nice to have a separate load balancer for updates to
> split
> >> out query and update load monitoring and alerting.
> >>
> >> With a smart load balancer, you could send the same query back to the
> same
> >> host, but AWS load balancers aren’t very smart.
> >>
> >> wunder
> >> Walter Underwood
> >> wunder@wunderwood.org
> >> http://observer.wunderwood.org/ (my blog)
> >>
> >>> On Dec 13, 2022, at 3:50 AM, Dave <ha...@gmail.com>
> wrote:
> >>>
> >>> Ha I meant qtimes not atone. Also in general you shouldn’t use a load
> >> balancer with solr, since you won’t be able to keep the index hot and n
> >> memory for each subsequent query if you are paging through results. The
> >> best way in my experience is to have failovers for your nodes, instead
> of
> >> load balancing.
> >>>
> >>>> On Dec 13, 2022, at 12:13 AM, Shawn Heisey <ap...@elyograg.org>
> wrote:
> >>>>
> >>>> On 12/12/22 13:14, Pradeep wrote:
> >>>>> How to check time taken by solr to execute api? Also can you share me
> >> solr
> >>>>> doc how we can query manually solr index for specific record or any
> >> api's
> >>>>> you can provide.
> >>>>
> >>>> Not entirely sure what you are asking here. I will try to answer what
> >> I can discern.
> >>>>
> >>>> In solr.log, each query is logged if you don't change the default
> >> logging levels. Each of those log lines will include a qtime parameter,
> >> counting the number of milliseconds it took to execute the query. The
> >> amount of time it takes to build the response and send it over the
> network
> >> is not included in the qtime.
> >>>>
> >>>> I don't know anything about your index, so I wouldn't be able to give
> >> you explicit instructions for querying for a specific document. But in
> >> most cases a query string like "id:value" will return a specific
> document,
> >> assuming that the "id" field is your uniqueKey.
> >>>>
> >>>> Thanks,
> >>>> Shawn
> >>>>
> >>
> >>
>
>
Re: Slowness in Solr Optimize
Posted by Walter Underwood <wu...@wunderwood.org>.
Sticky sessions only help for later pages. They don’t direct new queries to servers that already have that query cached. That is a bigger win than second pages.
I’ve thought about making a hash of the query-meaningful params (q, bq, …) and routing based on that, but that has other problems. When one server goes down, you simultaneously blow out every cache, which can cause cascading server overload. This is an actual problem, that took down the Infoseek search engine years ago. All the servers went down like dominos.
There are some hashing approaches to fix this, but testing gets difficult.
If you want really effective caching, put an HTTP cache in front of the load balancer. That is way better than fussing with request routing and the testing is minimal, because cache servers are designed to do exactly that. I did some benchmarks, and Varnish caching was slightly faster than the caching in a single Solr server.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Dec 13, 2022, at 11:27 AM, David Hastings <ha...@gmail.com> wrote:
>
> Ah, that makes sense. If you can do sticky sessions and such with your
> balancers, plus I never had to deal with the throughput of something like
> Netflix, so for mine and most use cases, I still feel one very hot server
> is better than N warm ones.
>
> "but AWS load balancers aren’t very smart." - agreed, but it looks like
> they attempted something for it:
> https://docs.aws.amazon.com/elasticloadbalancing/latest/application/sticky-sessions.html
>
> but can you sticky session an app server to a solr server?
> Like
> user->loadbalancer->app server X->solr load blancer->solr server X
> and make sure that app server X- and solr server X stay connected for the
> user's session? would love to see how to configure this
>
>
> On Tue, Dec 13, 2022 at 1:53 PM Walter Underwood <wu...@wunderwood.org>
> wrote:
>
>> I’ve always run load balancers, starting with Solr 1.2 at Netflix.
>> Failover (cold) spares have cold caches, so have slow performance until the
>> cache fills. I configure N+1 capacity, where N servers can handle the
>> expected load, then we add one for failure handling. All the spares are hot.
>>
>> I even run updates to Solr Cloud through a load balancer. It is easy to
>> configure and Solr is very efficient at forwarding documents to shard
>> leaders. It is nice to have a separate load balancer for updates to split
>> out query and update load monitoring and alerting.
>>
>> With a smart load balancer, you could send the same query back to the same
>> host, but AWS load balancers aren’t very smart.
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>>
>>> On Dec 13, 2022, at 3:50 AM, Dave <ha...@gmail.com> wrote:
>>>
>>> Ha I meant qtimes not atone. Also in general you shouldn’t use a load
>> balancer with solr, since you won’t be able to keep the index hot and n
>> memory for each subsequent query if you are paging through results. The
>> best way in my experience is to have failovers for your nodes, instead of
>> load balancing.
>>>
>>>> On Dec 13, 2022, at 12:13 AM, Shawn Heisey <ap...@elyograg.org> wrote:
>>>>
>>>> On 12/12/22 13:14, Pradeep wrote:
>>>>> How to check time taken by solr to execute api? Also can you share me
>> solr
>>>>> doc how we can query manually solr index for specific record or any
>> api's
>>>>> you can provide.
>>>>
>>>> Not entirely sure what you are asking here. I will try to answer what
>> I can discern.
>>>>
>>>> In solr.log, each query is logged if you don't change the default
>> logging levels. Each of those log lines will include a qtime parameter,
>> counting the number of milliseconds it took to execute the query. The
>> amount of time it takes to build the response and send it over the network
>> is not included in the qtime.
>>>>
>>>> I don't know anything about your index, so I wouldn't be able to give
>> you explicit instructions for querying for a specific document. But in
>> most cases a query string like "id:value" will return a specific document,
>> assuming that the "id" field is your uniqueKey.
>>>>
>>>> Thanks,
>>>> Shawn
>>>>
>>
>>
Re: Slowness in Solr Optimize
Posted by David Hastings <ha...@gmail.com>.
Ah, that makes sense. If you can do sticky sessions and such with your
balancers, plus I never had to deal with the throughput of something like
Netflix, so for mine and most use cases, I still feel one very hot server
is better than N warm ones.
"but AWS load balancers aren’t very smart." - agreed, but it looks like
they attempted something for it:
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/sticky-sessions.html
but can you sticky session an app server to a solr server?
Like
user->loadbalancer->app server X->solr load blancer->solr server X
and make sure that app server X- and solr server X stay connected for the
user's session? would love to see how to configure this
On Tue, Dec 13, 2022 at 1:53 PM Walter Underwood <wu...@wunderwood.org>
wrote:
> I’ve always run load balancers, starting with Solr 1.2 at Netflix.
> Failover (cold) spares have cold caches, so have slow performance until the
> cache fills. I configure N+1 capacity, where N servers can handle the
> expected load, then we add one for failure handling. All the spares are hot.
>
> I even run updates to Solr Cloud through a load balancer. It is easy to
> configure and Solr is very efficient at forwarding documents to shard
> leaders. It is nice to have a separate load balancer for updates to split
> out query and update load monitoring and alerting.
>
> With a smart load balancer, you could send the same query back to the same
> host, but AWS load balancers aren’t very smart.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
> > On Dec 13, 2022, at 3:50 AM, Dave <ha...@gmail.com> wrote:
> >
> > Ha I meant qtimes not atone. Also in general you shouldn’t use a load
> balancer with solr, since you won’t be able to keep the index hot and n
> memory for each subsequent query if you are paging through results. The
> best way in my experience is to have failovers for your nodes, instead of
> load balancing.
> >
> >> On Dec 13, 2022, at 12:13 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> >>
> >> On 12/12/22 13:14, Pradeep wrote:
> >>> How to check time taken by solr to execute api? Also can you share me
> solr
> >>> doc how we can query manually solr index for specific record or any
> api's
> >>> you can provide.
> >>
> >> Not entirely sure what you are asking here. I will try to answer what
> I can discern.
> >>
> >> In solr.log, each query is logged if you don't change the default
> logging levels. Each of those log lines will include a qtime parameter,
> counting the number of milliseconds it took to execute the query. The
> amount of time it takes to build the response and send it over the network
> is not included in the qtime.
> >>
> >> I don't know anything about your index, so I wouldn't be able to give
> you explicit instructions for querying for a specific document. But in
> most cases a query string like "id:value" will return a specific document,
> assuming that the "id" field is your uniqueKey.
> >>
> >> Thanks,
> >> Shawn
> >>
>
>
Re: Slowness in Solr Optimize
Posted by Walter Underwood <wu...@wunderwood.org>.
I’ve always run load balancers, starting with Solr 1.2 at Netflix. Failover (cold) spares have cold caches, so have slow performance until the cache fills. I configure N+1 capacity, where N servers can handle the expected load, then we add one for failure handling. All the spares are hot.
I even run updates to Solr Cloud through a load balancer. It is easy to configure and Solr is very efficient at forwarding documents to shard leaders. It is nice to have a separate load balancer for updates to split out query and update load monitoring and alerting.
With a smart load balancer, you could send the same query back to the same host, but AWS load balancers aren’t very smart.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Dec 13, 2022, at 3:50 AM, Dave <ha...@gmail.com> wrote:
>
> Ha I meant qtimes not atone. Also in general you shouldn’t use a load balancer with solr, since you won’t be able to keep the index hot and n memory for each subsequent query if you are paging through results. The best way in my experience is to have failovers for your nodes, instead of load balancing.
>
>> On Dec 13, 2022, at 12:13 AM, Shawn Heisey <ap...@elyograg.org> wrote:
>>
>> On 12/12/22 13:14, Pradeep wrote:
>>> How to check time taken by solr to execute api? Also can you share me solr
>>> doc how we can query manually solr index for specific record or any api's
>>> you can provide.
>>
>> Not entirely sure what you are asking here. I will try to answer what I can discern.
>>
>> In solr.log, each query is logged if you don't change the default logging levels. Each of those log lines will include a qtime parameter, counting the number of milliseconds it took to execute the query. The amount of time it takes to build the response and send it over the network is not included in the qtime.
>>
>> I don't know anything about your index, so I wouldn't be able to give you explicit instructions for querying for a specific document. But in most cases a query string like "id:value" will return a specific document, assuming that the "id" field is your uniqueKey.
>>
>> Thanks,
>> Shawn
>>
Re: Slowness in Solr Optimize
Posted by Dave <ha...@gmail.com>.
Ha I meant qtimes not atone. Also in general you shouldn’t use a load balancer with solr, since you won’t be able to keep the index hot and n memory for each subsequent query if you are paging through results. The best way in my experience is to have failovers for your nodes, instead of load balancing.
> On Dec 13, 2022, at 12:13 AM, Shawn Heisey <ap...@elyograg.org> wrote:
>
> On 12/12/22 13:14, Pradeep wrote:
>> How to check time taken by solr to execute api? Also can you share me solr
>> doc how we can query manually solr index for specific record or any api's
>> you can provide.
>
> Not entirely sure what you are asking here. I will try to answer what I can discern.
>
> In solr.log, each query is logged if you don't change the default logging levels. Each of those log lines will include a qtime parameter, counting the number of milliseconds it took to execute the query. The amount of time it takes to build the response and send it over the network is not included in the qtime.
>
> I don't know anything about your index, so I wouldn't be able to give you explicit instructions for querying for a specific document. But in most cases a query string like "id:value" will return a specific document, assuming that the "id" field is your uniqueKey.
>
> Thanks,
> Shawn
>
Re: Slowness in Solr Optimize
Posted by Shawn Heisey <ap...@elyograg.org>.
On 12/12/22 13:14, Pradeep wrote:
> How to check time taken by solr to execute api? Also can you share me solr
> doc how we can query manually solr index for specific record or any api's
> you can provide.
Not entirely sure what you are asking here. I will try to answer what I
can discern.
In solr.log, each query is logged if you don't change the default
logging levels. Each of those log lines will include a qtime parameter,
counting the number of milliseconds it took to execute the query. The
amount of time it takes to build the response and send it over the
network is not included in the qtime.
I don't know anything about your index, so I wouldn't be able to give
you explicit instructions for querying for a specific document. But in
most cases a query string like "id:value" will return a specific
document, assuming that the "id" field is your uniqueKey.
Thanks,
Shawn
Re: Slowness in Solr Optimize
Posted by Pradeep <ku...@gmail.com>.
Thank you shawn.
I will have look, I have checked the solr.log file but didn't see any
error, after load balancer changes we are seeing this issue.
How to check time taken by solr to execute api? Also can you share me solr
doc how we can query manually solr index for specific record or any api's
you can provide.
Thanks
On Mon, 12 Dec, 2022, 18:51 Shawn Heisey, <el...@elyograg.org> wrote:
> On 12/12/22 08:14, Pradeep wrote:
> > We have migrated Apache solr aws load balancer from classic to
> Application
> > load balancer. We havent changed anything in solr other than migrating
> load
> > balancer but solr taking more time while committing in Solr.
> >
> > Fyi, we are processing 200 records every batch, this is failing at last
> > stage of commit in solr (deleting stale records in solr). But same logic
> > working fine from many years in classic load balancer but after migration
> > to ALB this is happenning.
>
> The message subject mentions optimize but the message body does not.
>
> Is optimize involved? An optimize of a large index is always going to
> be slow.
>
> Solr does not have any kind of "deleting stale records" that I have ever
> heard of, so that must be something your code, which is more difficult
> for us to troubleshoot because we are not going to have any idea how to
> get that code to give useful logs without a lot more information about
> that code.
>
> If you're encountering an error from Solr then solr.log should have at
> least some info about what happened and that might be enough info to
> figure out the problem.
>
> If changing the load balancer caused issues, then you may need to talk
> to someone who understands how to troubleshoot the LB.
>
> Basically we need a lot more information about what's happening to have
> any hope of helping you.
>
> Thanks,
> Shawn
>
>
Re: Slowness in Solr Optimize
Posted by Dave <ha...@gmail.com>.
Sounds like you should contact aws about it since it’s not a solr issue if the qtimes haven’t increased in the solr logs. And again, don’t load balance but that’s my personal opinion
> On Dec 13, 2022, at 6:50 AM, Pradeep <ku...@gmail.com> wrote:
>
> Hi,
>
> I cant change it to NLB at this moment, firstly why timeout we need to
> understand no clue at this moment. It works fine if i increase timeout from
> 60 secs to 4000 secs. Its same code working in classic load balancer but
> with ALB we have this issue.
>
>
> Thanks,
> Pradeep
>
>> On Tue, 13 Dec, 2022, 11:08 Deepak Goel, <de...@gmail.com> wrote:
>>
>> Can you try 'Network Load Balancer" in aws?
>>
>>
>> Deepak
>> "The greatness of a nation can be judged by the way its animals are treated
>> - Mahatma Gandhi"
>>
>> +91 73500 12833
>> deicool@gmail.com
>>
>> Facebook: https://www.facebook.com/deicool
>> LinkedIn: www.linkedin.com/in/deicool
>>
>> "Plant a Tree, Go Green"
>>
>> Make In India : http://www.makeinindia.com/home
>>
>>
>> On Tue, Dec 13, 2022 at 12:21 AM Shawn Heisey <el...@elyograg.org>
>> wrote:
>>
>>> On 12/12/22 08:14, Pradeep wrote:
>>>> We have migrated Apache solr aws load balancer from classic to
>>> Application
>>>> load balancer. We havent changed anything in solr other than migrating
>>> load
>>>> balancer but solr taking more time while committing in Solr.
>>>>
>>>> Fyi, we are processing 200 records every batch, this is failing at last
>>>> stage of commit in solr (deleting stale records in solr). But same
>> logic
>>>> working fine from many years in classic load balancer but after
>> migration
>>>> to ALB this is happenning.
>>>
>>> The message subject mentions optimize but the message body does not.
>>>
>>> Is optimize involved? An optimize of a large index is always going to
>>> be slow.
>>>
>>> Solr does not have any kind of "deleting stale records" that I have ever
>>> heard of, so that must be something your code, which is more difficult
>>> for us to troubleshoot because we are not going to have any idea how to
>>> get that code to give useful logs without a lot more information about
>>> that code.
>>>
>>> If you're encountering an error from Solr then solr.log should have at
>>> least some info about what happened and that might be enough info to
>>> figure out the problem.
>>>
>>> If changing the load balancer caused issues, then you may need to talk
>>> to someone who understands how to troubleshoot the LB.
>>>
>>> Basically we need a lot more information about what's happening to have
>>> any hope of helping you.
>>>
>>> Thanks,
>>> Shawn
>>>
>>>
>>
Re: Slowness in Solr Optimize
Posted by Pradeep <ku...@gmail.com>.
Hi,
I cant change it to NLB at this moment, firstly why timeout we need to
understand no clue at this moment. It works fine if i increase timeout from
60 secs to 4000 secs. Its same code working in classic load balancer but
with ALB we have this issue.
Thanks,
Pradeep
On Tue, 13 Dec, 2022, 11:08 Deepak Goel, <de...@gmail.com> wrote:
> Can you try 'Network Load Balancer" in aws?
>
>
> Deepak
> "The greatness of a nation can be judged by the way its animals are treated
> - Mahatma Gandhi"
>
> +91 73500 12833
> deicool@gmail.com
>
> Facebook: https://www.facebook.com/deicool
> LinkedIn: www.linkedin.com/in/deicool
>
> "Plant a Tree, Go Green"
>
> Make In India : http://www.makeinindia.com/home
>
>
> On Tue, Dec 13, 2022 at 12:21 AM Shawn Heisey <el...@elyograg.org>
> wrote:
>
> > On 12/12/22 08:14, Pradeep wrote:
> > > We have migrated Apache solr aws load balancer from classic to
> > Application
> > > load balancer. We havent changed anything in solr other than migrating
> > load
> > > balancer but solr taking more time while committing in Solr.
> > >
> > > Fyi, we are processing 200 records every batch, this is failing at last
> > > stage of commit in solr (deleting stale records in solr). But same
> logic
> > > working fine from many years in classic load balancer but after
> migration
> > > to ALB this is happenning.
> >
> > The message subject mentions optimize but the message body does not.
> >
> > Is optimize involved? An optimize of a large index is always going to
> > be slow.
> >
> > Solr does not have any kind of "deleting stale records" that I have ever
> > heard of, so that must be something your code, which is more difficult
> > for us to troubleshoot because we are not going to have any idea how to
> > get that code to give useful logs without a lot more information about
> > that code.
> >
> > If you're encountering an error from Solr then solr.log should have at
> > least some info about what happened and that might be enough info to
> > figure out the problem.
> >
> > If changing the load balancer caused issues, then you may need to talk
> > to someone who understands how to troubleshoot the LB.
> >
> > Basically we need a lot more information about what's happening to have
> > any hope of helping you.
> >
> > Thanks,
> > Shawn
> >
> >
>
Re: Slowness in Solr Optimize
Posted by Deepak Goel <de...@gmail.com>.
Can you try 'Network Load Balancer" in aws?
Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
Make In India : http://www.makeinindia.com/home
On Tue, Dec 13, 2022 at 12:21 AM Shawn Heisey <el...@elyograg.org> wrote:
> On 12/12/22 08:14, Pradeep wrote:
> > We have migrated Apache solr aws load balancer from classic to
> Application
> > load balancer. We havent changed anything in solr other than migrating
> load
> > balancer but solr taking more time while committing in Solr.
> >
> > Fyi, we are processing 200 records every batch, this is failing at last
> > stage of commit in solr (deleting stale records in solr). But same logic
> > working fine from many years in classic load balancer but after migration
> > to ALB this is happenning.
>
> The message subject mentions optimize but the message body does not.
>
> Is optimize involved? An optimize of a large index is always going to
> be slow.
>
> Solr does not have any kind of "deleting stale records" that I have ever
> heard of, so that must be something your code, which is more difficult
> for us to troubleshoot because we are not going to have any idea how to
> get that code to give useful logs without a lot more information about
> that code.
>
> If you're encountering an error from Solr then solr.log should have at
> least some info about what happened and that might be enough info to
> figure out the problem.
>
> If changing the load balancer caused issues, then you may need to talk
> to someone who understands how to troubleshoot the LB.
>
> Basically we need a lot more information about what's happening to have
> any hope of helping you.
>
> Thanks,
> Shawn
>
>
Re: Slowness in Solr Optimize
Posted by Shawn Heisey <el...@elyograg.org>.
On 12/12/22 08:14, Pradeep wrote:
> We have migrated Apache solr aws load balancer from classic to Application
> load balancer. We havent changed anything in solr other than migrating load
> balancer but solr taking more time while committing in Solr.
>
> Fyi, we are processing 200 records every batch, this is failing at last
> stage of commit in solr (deleting stale records in solr). But same logic
> working fine from many years in classic load balancer but after migration
> to ALB this is happenning.
The message subject mentions optimize but the message body does not.
Is optimize involved? An optimize of a large index is always going to
be slow.
Solr does not have any kind of "deleting stale records" that I have ever
heard of, so that must be something your code, which is more difficult
for us to troubleshoot because we are not going to have any idea how to
get that code to give useful logs without a lot more information about
that code.
If you're encountering an error from Solr then solr.log should have at
least some info about what happened and that might be enough info to
figure out the problem.
If changing the load balancer caused issues, then you may need to talk
to someone who understands how to troubleshoot the LB.
Basically we need a lot more information about what's happening to have
any hope of helping you.
Thanks,
Shawn