Re: Add dynamic field to existing index slow

[derrick cui <de...@yahoo.ca.INVALID>]

I have tested the query desperately, actually executing query is pretty fast, it only took a few minutes to go through all results including converting solr document to java object. So I believe the slowness is in persistence end.  BTW,  I am using linux system.


Sent from Yahoo Mail for iPhone


On Sunday, June 30, 2019, 4:52 PM, Shawn Heisey <ap...@elyograg.org> wrote:

On 6/30/2019 2:08 PM, derrick cui wrote:
> Good point Erick, I will try it today, but I have already use cursorMark in my query for deep pagination.
> Also I noticed that my cpu usage is pretty high, 8 cores, usage is over 700%. I am not sure it will help if I use ssd disk

That depends on whether the load is caused by iowait or by actual CPU usage.

If it's caused by iowait, then SSD would help, but additional memory 
would help more.  Retrieving data from the OS disk cache (which exists 
in main memory) is faster than SSD.

If it is actual CPU load, then it will take some additional poking 
around to figure out which part of your activities causes the load, as 
Erick mentioned.

It's normally a little bit easier to learn these things from Unix-like 
operating systems than from Windows.  What OS are you running Solr on?

Thanks,
Shawn

Re: Add dynamic field to existing index slow

[Erick Erickson <er...@gmail.com>]

OK, then let’s see the indexing code. Make sure you don’t
1> commit after every batch
2> never, never, never optimize.

BTW, you do not want to turn off commits entirely, there are some internal data structures that grow between commits. So I might do something like specify commitWithin on my adds for something like 5 minutes.

Best,
Erick

> On Jul 2, 2019, at 6:24 AM, derrick cui <de...@yahoo.ca.INVALID> wrote:
> 
> I have tested the query desperately, actually executing query is pretty fast, it only took a few minutes to go through all results including converting solr document to java object. So I believe the slowness is in persistence end.  BTW,  I am using linux system.
> 
> 
> Sent from Yahoo Mail for iPhone
> 
> 
> On Sunday, June 30, 2019, 4:52 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> 
> On 6/30/2019 2:08 PM, derrick cui wrote:
>> Good point Erick, I will try it today, but I have already use cursorMark in my query for deep pagination.
>> Also I noticed that my cpu usage is pretty high, 8 cores, usage is over 700%. I am not sure it will help if I use ssd disk
> 
> That depends on whether the load is caused by iowait or by actual CPU usage.
> 
> If it's caused by iowait, then SSD would help, but additional memory 
> would help more.  Retrieving data from the OS disk cache (which exists 
> in main memory) is faster than SSD.
> 
> If it is actual CPU load, then it will take some additional poking 
> around to figure out which part of your activities causes the load, as 
> Erick mentioned.
> 
> It's normally a little bit easier to learn these things from Unix-like 
> operating systems than from Windows.  What OS are you running Solr on?
> 
> Thanks,
> Shawn
> 
> 
>