You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by mganeshs <mg...@live.in> on 2018/03/26 16:45:27 UTC
Default Index config
Hi,
I haven't changed the solr config wrt index config, which means it's all
commented in the solrconfig.xml.
It's something like what I pasted before. But I would like to know whats the
default value of each of this.
Coz.. after loading to 6.5.1 and our document size also crossed 5GB in each
of our collection. Now update of document is taking time. So would like to
know whether we need to change any default configurations.
<indexConfig>
<lockType>${solr.lock.type:native}</lockType>
</indexConfig>
Advice...
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Default Index config
Posted by mganeshs <mg...@live.in>.
To add it further, in 6.5.1, while indexing... even sometimes one of solr
node goes down for a while and comes up automatically. During those period
all our calls to index fails. Even in the Solr admin UI, we can see node not
being active for a while and coming up again.
All these happens in 4 core machine ( r4-xlarge). If we move to r4-2xlarge (
8 core machine )... everything goes smooth without any issue. Even CPU also
stays with 50%.
Is that mean that we need 8 core machine to run the index and query with
such a data and heavy indexing rate ?
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Default Index config
Posted by mganeshs <mg...@live.in>.
Hi Shawn,
We found following link
<https://www.google.com/url?q=http://lucene.472066.n3.nabble.com/CommitScheduler-Thread-blocked-due-to-excessive-number-of-Merging-Threads-td4353964.html&sa=D&source=hangouts&ust=1523527480128000&usg=AFQjCNENaNhq49nMSj8WtS_Qx-N4K23v0Q>
where its mentioned like in 6.2.1 it's faster where as in 6.6 its slower.
Keep this, we too tried with 6.2.1 in our performance environment and we
found that CPU usage came down to 60 to 70% where as in 6.5.1 it was always
more than 95%
Settings are same and data size and indexing speed remains same. Pls check
the JVM snapshot
<https://lh3.googleusercontent.com/-MHYsop5Vovo/Ws3V8KrT2VI/AAAAAAAAALI/GXmcWA4OtPQwwpF09scU5riJ1VHFOUtHQCL0BGAYYCw/h1080/2018-04-11.png>
when we index using 6.2.1
Following is the snapshot
<https://lh3.googleusercontent.com/-FoOmxYACbKc/Ws3gUQ323rI/AAAAAAAACeA/xdZedHlr1pcRbBrxhNJFPSvZqaH6rIoEACL0BGAYYCw/h363/2018-04-11.png>
taken with 6.5.1
Is there any reason why such a huge difference with CPU usage patterns
between 6.2.1 and 6.5.1 ?
Can we do something in 6.5.1 to make it as 6.2.1? Because we don't want to
downgrade to 6.2.1 from 6.5.1.
Let us know your thoughts on this.
Thanks and Regards,
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Default Index config
Posted by mganeshs <mg...@live.in>.
Hi Shawn,
Thanks for the reply.
Yes we use only one solr client. Though collection name is passed in the
function, we are using same client for now.
Regarding merge config, after reading lot of forums and listening to
presentation of revolution 2017, idea is to reduce the merge frequency, so
that CPU usage pattern will come down from 100 to 70% for a while and only
when merges happens it will go to 100% ( where as now it's always above 95%
) which we see it as not a good sign of CPU always more than 95% since we
run other components as well in this server. So to reduce the merge
frequency, i was trying that.
Thanks for sharing your config, will try to check with that too and post you
an update on the result.
Thanks and Regards,
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Default Index config
Posted by Shawn Heisey <ap...@elyograg.org>.
On 4/9/2018 4:04 AM, mganeshs wrote:
> Regarding CPU high, when we are troubleshooting, we found that Merge threads
> are keep on running and it's take most CPU time ( as per Visual JVM ).
With a one second autoSoftCommit, nearly constant indexing will produce
a lot of very small index segments. Those index segments will have to
be merged eventually. You have increased the merge policy numbers which
will reduce the total number of merges, but each merge is going to be
larger than it would with defaults, so it's going to take a little bit
longer. This isn't too big a deal with first-level merges, but at the
higher levels, they do get large -- no matter what the configuration is.
> *Note*: following is the code snippet we use for indexing / adding solr
> document in batch per collection
>
> /for (SolrCollectionList solrCollection : SolrCollectionList.values()) {
> CollectionBucket collectionBucket = getCollectionBucket(solrCollection);
> List<SolrInputDocument> solrInputDocuments =
> collectionBucket.getSolrInputDocumentList();
> String collectionName = collectionBucket.getCollectionName();
> try {
> if(solrInputDocuments.size() > 0) {
> CloudSolrClient solrClient =
> PlatformIndexManager.getInstance().getCloudSolrClient(collectionName);
> solrClient.add(collectionName, solrInputDocuments);
> }
> }/
>
> *where solrClient is created as below
> *
> /this.cloudSolrClient = new
> CloudSolrClient.Builder().withZkHost(zooKeeperHost).withHttpClient(HttpClientUtil.HttpClientFactory.createHttpClient()).build();
> this.cloudSolrClient.setZkClientTimeout(30000);
> /
Is that code running on the Solr server, or on a different machine? Are
you creating a SolrClient each time you use it, or have you created
client objects that get re-used?
You don't need a different SolrClient object for each collection. Your
"getCloudSolrClient" method takes a collection name, which suggests
there might be a different client object for each one. Most of the
time, you need precisely one client object for the entire application.
> Hard commit is kept as automatic and set to 15000 ms.
> In this process, we also see, when merge is happening, and already
> maxMergeCount ( default one ) is reached, commits are getting delayed and
> solrj client ( where we add document ) is getting blocked and once once of
> Merge thread process the merge, then solrj client returns the result.
> How do we avoid this blocking of solrj client ? Do I need to go out of
> default config for this scenario? I mean change the merge factor
> configuration ?
>
> Can you suggest what would be merge config for such a scenario ? Based on
> forums, I tried to change the merge settings to the following,
What are you trying to accomplish by changing the merge policy? It's
fine to find information for a config on the Internet, but you need to
know what that config *does* before you use it, and make sure it aligns
with your goals. On mine, I change maxMergeAtOnce and segmentsPerTier
to 35, and maxMergeAtOnceExplicit to 105. I know exactly what I'm
trying to do with this config -- reduce the frequency of merges. Each
merge is going to be larger with this config, but they will happen less
frequently. These three settings are the only ones that I change in my
merge policy. Changing all of the other settings that you have changed
should not be necessary. I make one other adjustment in this area -- to
the merge scheduler.
> In same solr node, we have multiple index / collection. In that case,
> whether TieredMergePolicyFactory will be right option or for multiple
> collection in same node we should go for other merge policy ( like LogByte
> etc )
TieredMergePolicy was made the default policy after a great deal of
testing and discussion by Lucene developers. They found that it works
better than the others for the vast majority of users. It is likely the
best choice for you too.
These are the settings that I use in indexConfig to reduce the impact of
merges on my indexing:
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
<int name="maxMergeAtOnce">35</int>
<int name="segmentsPerTier">35</int>
<int name="maxMergeAtOnceExplicit">105</int>
</mergePolicy>
<mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
<int name="maxThreadCount">1</int>
<int name="maxMergeCount">6</int>
</mergeScheduler>
Note that this config is designed for 6.x and earlier. I do not know if
it will work in 7.x. It probably needs to be adjusted to the new
Factory config. You can use it as a guide, though.
Thanks,
Shawn
Re: Default Index config
Posted by mganeshs <mg...@live.in>.
Hi Shawn,
Regarding CPU high, when we are troubleshooting, we found that Merge threads
are keep on running and it's take most CPU time ( as per Visual JVM ). GC is
not causing any issue as we use the default GC and also tried with G1 as you
suggested over here
<https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning_for_Solr>
Though it's only background process, we are suspecting whether it's causing
CPU to go high.
Since we are using SOLR as real time indexing of data and depending on its
result immd. to show it in UI as well. So we keep adding document around 100
to 200 documents in parallel in a sec. Also it would be in batch of 20 solr
documents list in one add...
*Note*: following is the code snippet we use for indexing / adding solr
document in batch per collection
/for (SolrCollectionList solrCollection : SolrCollectionList.values()) {
CollectionBucket collectionBucket = getCollectionBucket(solrCollection);
List<SolrInputDocument> solrInputDocuments =
collectionBucket.getSolrInputDocumentList();
String collectionName = collectionBucket.getCollectionName();
try {
if(solrInputDocuments.size() > 0) {
CloudSolrClient solrClient =
PlatformIndexManager.getInstance().getCloudSolrClient(collectionName);
solrClient.add(collectionName, solrInputDocuments);
}
}/
*where solrClient is created as below
*
/this.cloudSolrClient = new
CloudSolrClient.Builder().withZkHost(zooKeeperHost).withHttpClient(HttpClientUtil.HttpClientFactory.createHttpClient()).build();
this.cloudSolrClient.setZkClientTimeout(30000);
/
Hard commit is kept as automatic and set to 15000 ms.
In this process, we also see, when merge is happening, and already
maxMergeCount ( default one ) is reached, commits are getting delayed and
solrj client ( where we add document ) is getting blocked and once once of
Merge thread process the merge, then solrj client returns the result.
How do we avoid this blocking of solrj client ? Do I need to go out of
default config for this scenario? I mean change the merge factor
configuration ?
Can you suggest what would be merge config for such a scenario ? Based on
forums, I tried to change the merge settings to the following,
<mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
<int name="maxMergeAtOnce">30</int>
<int name="maxMergeAtOnceExplicit">30</int>
<int name="segmentsPerTier">30</int>
<int name="floorSegmentMB">2048</int>
<int name="maxMergedSegmentMB">512</int>
<double name="noCFSRatio">0.1</double>
<int name="maxCFSSegmentSizeMB">2048</int>
<double name="reclaimDeletesWeight">2.0</double>
<double name="forceMergeDeletesPctAllowed">10.0</double>
</mergePolicyFactory>
But couldn't see any much change in the behaviour.
In same solr node, we have multiple index / collection. In that case,
whether TieredMergePolicyFactory will be right option or for multiple
collection in same node we should go for other merge policy ( like LogByte
etc )
Can you throw some light on this aspects ?
Regards,
Regarding auto commit, we discussed lot with our product owners and atlast
> we are forced to keep it to 1sec and we couldn't increase further. As this
> itself, sometimes our customers says that they have to refresh their pages
> for couple of times to get the update from solr. So we can't increase
> further.
I understand pressure from nontechnical departments for very low
response times. Executives, sales, and marketing are usually the ones
making those kinds of demands. I think you should push back on that
particular requirement on technical grounds.
A soft commit interval that low *can* contribute to performance issues.
It doesn't always cause them, I'm just saying that it *can*. Maybe
increasing it to five or ten seconds could help performance, or maybe it
will make no real difference at all.
> Yes. As of now only solr is running in that machine. But intially we were
> running along with hbase region servers and was working fine. But due to
> CPU
> spikes and OS disk cache, we are forced to move solr to separate machine.
> But just I checked, our solr data folder size is coming only to 17GB. 2
> collection has around 5GB and other are have 2 to 3 GB of size. If you say
> that only 2/3 of total size comes to OS disk cache, in top command VIRT
> property it's always 28G, which means more than what we have. Why is
> that...
> Pls check that top command & GC we used in this doc
> <https://docs.google.com/document/d/1SaKPbGAKEPP8bSbdvfX52gaLsYWnQfDqfmV802hWIiQ/edit?usp=sharing>
The VIRT memory should be about equivalent to the RES size plus the size
of all the index data on the system. So that looks about right. The
actual amount of memory allocated by Java for the heap and other memory
structures is approximately equal to RES minus SHR.
I am not sure whether the SHR size gets counted in VIRT. It probably
does. On some Linux systems, SHR grows to a very high number, but when
that happens, it typically doesn't reflect actual memory usage. I do
not know why this sometimes happens.That is a question for Oracle, since
they are the current owners of Java.
Only 5GB is in the buff/cache area. The system has 13GB of free
memory. That system is NOT low on memory.
With 4 CPUs, a load average in the 3-4 range is an indication that the
server is busy. I can't say for sure whether it means the server is
overloaded. Sometimes the load average on a system that's working well
can go higher than the CPU count, sometimes a load average well below
the CPU count is shown on a system with major performance issues. It's
difficult to say. The instantaneous CPU usage on the Solr process in
that screenshot is 384 percent. Which means that it is exercising the
CPUs hard. But this might be perfectly OK. 96.3 percent of the CPU is
being used by user processes, a VERY small amount is being used by
system, and the iowait percentage is zero. Typically servers that are
struggling will have a higher percentage in system and/or iowait, and I
don't see that here.
> Queries are quiet fast, most of time simple queries with fq. Regarding
> index, during peak hours, we index around 100 documents in a second in a
> average.
That's good. And not surprising, given how little memory pressure and
how much free memory there is. An indexing rate of 100 per second
doesn't seem like a lot of indexing to me, but for some indexes, it
might be very heavy. If your general performance is good, I wouldn't be
too concerned about it.
> Regarding release, initially we tried with 6.4.1 and since many
> discussions
> over here, mentioned like moving to 6.5.x will solve lot of performance
> issues etc, so we moved to 6.5.1. We will move to 6.6.3 in near future.
The 6.4.1 version had a really bad bug in it that killed performance for
most users. Some might not have even noticed a problem, though. It's
difficult to say for sure whether it would be something you would
notice, or whether you would see an increase in performance by upgrading.
> Hope I have given enough information. One strange thing is that, CPU and
> memory spike are not seen when we move to r4.xlarge to r4.2xlarge ( which
> is
> 8 core with 60 GB RAM ). But this would not be cost effective. What's
> making
> CPU and memory to go high in this new version ( due to doc values )? If I
> switch off docvalues will CPU & Memory spikes will get reduced ?
Overall memory usage (outside of the Java heap) looks great to me. CPU
usage is high, but I can't tell if it's TOO high. As a proof of concept,
I think you should try raising autoSoftCommit to five seconds. If
maxDocs is configured on either autoCommit or autoSoftCommit, remove it
so that only maxTime is there, regardless of whether you actually change
maxTime. If raising autoSoftCommit makes no real difference, then the 1
second autoSoftCommit probably isn't a worry. I bet if you raised it to
five seconds, most users would never notice anything different.
If you want to provide a GC log to us that covers a relatively long
timeframe, we can analyze that and let you know whether your heap is
sized appropriately, or whether it might be too big or too small, and
whether garbage collection pauses are keeping your CPU usage high. The
standard Solr startup in most current versions always logs GC activity.
It will usually be in the same directory as solr.log.
Do you know what typical and peak queries per second are on your Solr
servers? If your query rate is high, handling that will probably
require more servers and a higher replica count.
Thanks,
Shawn
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Default Index config
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/28/2018 9:44 PM, mganeshs wrote:
> Regarding auto commit, we discussed lot with our product owners and atlast
> we are forced to keep it to 1sec and we couldn't increase further. As this
> itself, sometimes our customers says that they have to refresh their pages
> for couple of times to get the update from solr. So we can't increase
> further.
I understand pressure from nontechnical departments for very low
response times. Executives, sales, and marketing are usually the ones
making those kinds of demands. I think you should push back on that
particular requirement on technical grounds.
A soft commit interval that low *can* contribute to performance issues.
It doesn't always cause them, I'm just saying that it *can*. Maybe
increasing it to five or ten seconds could help performance, or maybe it
will make no real difference at all.
> Yes. As of now only solr is running in that machine. But intially we were
> running along with hbase region servers and was working fine. But due to CPU
> spikes and OS disk cache, we are forced to move solr to separate machine.
> But just I checked, our solr data folder size is coming only to 17GB. 2
> collection has around 5GB and other are have 2 to 3 GB of size. If you say
> that only 2/3 of total size comes to OS disk cache, in top command VIRT
> property it's always 28G, which means more than what we have. Why is that...
> Pls check that top command & GC we used in this doc
> <https://docs.google.com/document/d/1SaKPbGAKEPP8bSbdvfX52gaLsYWnQfDqfmV802hWIiQ/edit?usp=sharing>
The VIRT memory should be about equivalent to the RES size plus the size
of all the index data on the system. So that looks about right. The
actual amount of memory allocated by Java for the heap and other memory
structures is approximately equal to RES minus SHR.
I am not sure whether the SHR size gets counted in VIRT. It probably
does. On some Linux systems, SHR grows to a very high number, but when
that happens, it typically doesn't reflect actual memory usage. I do
not know why this sometimes happens.That is a question for Oracle, since
they are the current owners of Java.
Only 5GB is in the buff/cache area. The system has 13GB of free
memory. That system is NOT low on memory.
With 4 CPUs, a load average in the 3-4 range is an indication that the
server is busy. I can't say for sure whether it means the server is
overloaded. Sometimes the load average on a system that's working well
can go higher than the CPU count, sometimes a load average well below
the CPU count is shown on a system with major performance issues. It's
difficult to say. The instantaneous CPU usage on the Solr process in
that screenshot is 384 percent. Which means that it is exercising the
CPUs hard. But this might be perfectly OK. 96.3 percent of the CPU is
being used by user processes, a VERY small amount is being used by
system, and the iowait percentage is zero. Typically servers that are
struggling will have a higher percentage in system and/or iowait, and I
don't see that here.
> Queries are quiet fast, most of time simple queries with fq. Regarding
> index, during peak hours, we index around 100 documents in a second in a
> average.
That's good. And not surprising, given how little memory pressure and
how much free memory there is. An indexing rate of 100 per second
doesn't seem like a lot of indexing to me, but for some indexes, it
might be very heavy. If your general performance is good, I wouldn't be
too concerned about it.
> Regarding release, initially we tried with 6.4.1 and since many discussions
> over here, mentioned like moving to 6.5.x will solve lot of performance
> issues etc, so we moved to 6.5.1. We will move to 6.6.3 in near future.
The 6.4.1 version had a really bad bug in it that killed performance for
most users. Some might not have even noticed a problem, though. It's
difficult to say for sure whether it would be something you would
notice, or whether you would see an increase in performance by upgrading.
> Hope I have given enough information. One strange thing is that, CPU and
> memory spike are not seen when we move to r4.xlarge to r4.2xlarge ( which is
> 8 core with 60 GB RAM ). But this would not be cost effective. What's making
> CPU and memory to go high in this new version ( due to doc values )? If I
> switch off docvalues will CPU & Memory spikes will get reduced ?
Overall memory usage (outside of the Java heap) looks great to me. CPU
usage is high, but I can't tell if it's TOO high. As a proof of concept,
I think you should try raising autoSoftCommit to five seconds. If
maxDocs is configured on either autoCommit or autoSoftCommit, remove it
so that only maxTime is there, regardless of whether you actually change
maxTime. If raising autoSoftCommit makes no real difference, then the 1
second autoSoftCommit probably isn't a worry. I bet if you raised it to
five seconds, most users would never notice anything different.
If you want to provide a GC log to us that covers a relatively long
timeframe, we can analyze that and let you know whether your heap is
sized appropriately, or whether it might be too big or too small, and
whether garbage collection pauses are keeping your CPU usage high. The
standard Solr startup in most current versions always logs GC activity.
It will usually be in the same directory as solr.log.
Do you know what typical and peak queries per second are on your Solr
servers? If your query rate is high, handling that will probably
require more servers and a higher replica count.
Thanks,
Shawn
Re: Default Index config
Posted by mganeshs <mg...@live.in>.
Hi Shawn,
Thanks again for detailed reply.
Regarding auto commit, we discussed lot with our product owners and atlast
we are forced to keep it to 1sec and we couldn't increase further. As this
itself, sometimes our customers says that they have to refresh their pages
for couple of times to get the update from solr. So we can't increase
further.
We have kept openSearcher, it's as typical configuration & recommended only.
Yes. As of now only solr is running in that machine. But intially we were
running along with hbase region servers and was working fine. But due to CPU
spikes and OS disk cache, we are forced to move solr to separate machine.
But just I checked, our solr data folder size is coming only to 17GB. 2
collection has around 5GB and other are have 2 to 3 GB of size. If you say
that only 2/3 of total size comes to OS disk cache, in top command VIRT
property it's always 28G, which means more than what we have. Why is that...
Pls check that top command & GC we used in this doc
<https://docs.google.com/document/d/1SaKPbGAKEPP8bSbdvfX52gaLsYWnQfDqfmV802hWIiQ/edit?usp=sharing>
Yes we are using solr cloud only.
Queries are quiet fast, most of time simple queries with fq. Regarding
index, during peak hours, we index around 100 documents in a second in a
average.
I also shared the CPU utilization in the same doc
<https://docs.google.com/document/d/1SaKPbGAKEPP8bSbdvfX52gaLsYWnQfDqfmV802hWIiQ/edit#heading=h.ahsgapiu4829>
Regarding release, initially we tried with 6.4.1 and since many discussions
over here, mentioned like moving to 6.5.x will solve lot of performance
issues etc, so we moved to 6.5.1. We will move to 6.6.3 in near future.
Hope I have given enough information. One strange thing is that, CPU and
memory spike are not seen when we move to r4.xlarge to r4.2xlarge ( which is
8 core with 60 GB RAM ). But this would not be cost effective. What's making
CPU and memory to go high in this new version ( due to doc values )? If I
switch off docvalues will CPU & Memory spikes will get reduced ?
Let me know...
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Default Index config
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/27/2018 9:35 PM, mganeshs wrote:
> I am using the default configuration, as all solr experts say default one
> suits for most of the cases and so following the defaults. We changed only
> the commits part and using 15000 for hard commits and 1 sec for soft commit.
> All other setting like locking, deleting policy, merge, directory, etc are
> left to default ones.
One second for autoSoftCommit has a tendency to cause a lot of issues.
That interval should be set as large as you can tolerate for change
visibility. I personally would want to see a setting there of at least
one minute, but I know that this is not fast enough for a lot of users.
Does autoCommit have openSearcher set to false? This is a typical
configuration, and recommended.
> One strange thing we noticed after moving from solr 5.x to solr 6.5.1 is
> that CPU and RAM usage is increased drastically. We have two solr nodes, one
> for data and another for replica. It's EC2 r4.xlarge machine. We have almost
> 6 collection and each carries around 5GB of data in average and in couple
> of collection we have frequent updates too.
That kind of instance has 30.5 GB of total memory. With 8GB assigned to
the heap, that leaves about 22GB for everything else. If Solr is the
only thing running on the machine, and your numbers mean that each
server has about 30GB of index data, then that means you can get about
two thirds of the index into the OS disk cache. Usually this is enough
for decent performance, but that's not the case for everyone. If this
is an accurate picture of your Solr install, I am not really worried
about total memory size. I can't make a guess about it unless I have
accurate information.
The way you're phrasing things, it sounds like you're running in
SolrCloud mode. Is that correct?
Since you're probably running Solr on Linux, can you get me a screenshot
from the "top" program? It must be specifically that program -- don't
use an alternate utility like "htop".
Run top, press shift-M to sort the display by memory, and then grab a
screenshot or photo of the display. Share that file somewhere and
provide a URL to access it. With that info, we can get a good picture
of system health. Be sure that when that "top" display is grabbed, that
the system is experiencing whatever problems you're trying to fix.
What do you know about your query load and how fast the data is being
indexed?
> In solr 5.x we didn't feel this much of RAM and CPU usages. CPU is always 80
> to 90% even if we are trying to index or update some 50 docs at one shot and
> RAM it occupies whatever we give. We started with 8GB of Heap. But its
> always 8GB. Initially we were using CMS GC and tried with G1 GC. Only
> difference is that, In case of CMS, even after starting solr, cpu goes to
> 80%, where as in G1, after started solr it's around 10% and when load comes
> ( around 100 to 200 docs in 5 mins ) it's goes 90% ( in both CMS and G1 )
The default garbage collection tuning does a pretty good job most of the
time. For some people, G1 works even better, but this can only happen
if it is tuned. Simply turning on G1GC and leaving it untuned will
probably get you WORSE performance than leaving the GC tuning at the
settings that the project has provided.
> When we tried to profile the solr process, we found that merging is keep on
> happening.
Can you elaborate more on exactly what you observed, and what
conclusions you came to based on that information?
> This spikes of CPU and memory, we are seeing only after moving to 6.5.1. Is
> that stable version ? moving to latest stable will solve this issue or we
> miss something wrt configurations ? Do we need to change the solr default
> config ?
Virtually all releases are considered stable at the time of release. If
they weren't, then they wouldn't be released! I'm only aware of two
times that versions were released without being called "stable." Those
were the 4.0-ALPHA and 4.0-BETA releases. There have been no ALPHA or
BETA releases since then.
Pretty much every version is later revealed to have bugs, and sometimes
those problems are big enough that the release could be called
unstable. But that is a determination made AFTER release, not before.
6.5.1 was announced on 27 April 2017. It is a bugfix release, mostly
identical to 6.5.0, which was announced exactly one month earlier, on 27
March 2017. Which makes it one year old.
There are a number of bugfixes in 6.5.1 compared to 6.5.0, but the two
are substantially similar.
http://mail-archives.apache.org/mod_mbox/www-announce/201704.mbox/%3CCAKUpjcQiYLZ1a+ZM=oHUxJywLj9vNub8q9b_iLrvmkidiDV2xg@mail.gmail.com%3E
If you're going to run 6.x, then you should run 6.6.3. The latest
version currently out is 7.2.1. Version 7.3.0 is just around the corner
-- release is underway right now. I haven't looked through the
changelogs, so I don't really know what's coming.
Thanks,
Shawn
Re: Default Index config
Posted by mganeshs <mg...@live.in>.
Hi Shawn,
Thanks for detail mail. Yes I am behind the IndexConfig only.
Regarding 5GB size of collection, it's not one document. It has almost 3M of
docs in that collection.
I am using the default configuration, as all solr experts say default one
suits for most of the cases and so following the defaults. We changed only
the commits part and using 15000 for hard commits and 1 sec for soft commit.
All other setting like locking, deleting policy, merge, directory, etc are
left to default ones.
One strange thing we noticed after moving from solr 5.x to solr 6.5.1 is
that CPU and RAM usage is increased drastically. We have two solr nodes, one
for data and another for replica. It's EC2 r4.xlarge machine. We have almost
6 collection and each carries around 5GB of data in average and in couple
of collection we have frequent updates too.
In solr 5.x we didn't feel this much of RAM and CPU usages. CPU is always 80
to 90% even if we are trying to index or update some 50 docs at one shot and
RAM it occupies whatever we give. We started with 8GB of Heap. But its
always 8GB. Initially we were using CMS GC and tried with G1 GC. Only
difference is that, In case of CMS, even after starting solr, cpu goes to
80%, where as in G1, after started solr it's around 10% and when load comes
( around 100 to 200 docs in 5 mins ) it's goes 90% ( in both CMS and G1 )
When we tried to profile the solr process, we found that merging is keep on
happening.
This spikes of CPU and memory, we are seeing only after moving to 6.5.1. Is
that stable version ? moving to latest stable will solve this issue or we
miss something wrt configurations ? Do we need to change the solr default
config ?
Advice...
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Default Index config
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/26/2018 10:45 AM, mganeshs wrote:
> I haven't changed the solr config wrt index config, which means it's all
> commented in the solrconfig.xml.
>
> It's something like what I pasted before. But I would like to know whats the
> default value of each of this.
Default value of *what* exactly? There are a LOT of possible settings
you could be asking about, it'll take too much time to research them all
and give you every possible default.
I can't tell what you're referring to when you mention "what I pasted
before." Later in your message I do see a mostly empty <indexConfig> tag.
If you're after defaults for indexConfig, the comments in Solr's example
config do a pretty good job of covering defaults. I think that each
commented config section *has* the default setting already in it. In a
download of the latest version, look for
server/solr/configsets/_default/conf/solrconfig.xml.
> Coz.. after loading to 6.5.1 and our document size also crossed 5GB in each
> of our collection. Now update of document is taking time. So would like to
> know whether we need to change any default configurations.
5GB for a *single* document? That sounds like a document that probably
shouldn't get indexed, because it might cause all sorts of problems.
If 5GB is the size of the entire index, then I can say that while this
is not exactly a *small* index, there are a LOT of indexes in the wild
that are MUCH bigger. I have such indexes in my Solr installs, and
while the performance of those indexes isn't lightning fast, it's pretty
good.
As your index size increases, Solr is going to slow down. That's just
how things work and cannot be changed. If the slowdown is extreme for a
small increase in size, that usually means that the system doesn't have
enough of some resource, usually memory. Additional hardware or
hardware upgrades may be required.
If you describe what you are seeing, and how that differs from what you
EXPECT to be seeing, then we might be able to get somewhere. More
information may be required to help. If that's the case, we will ask
for additional information.
Thanks,
Shawn