You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Zheng Lin Edwin Yeo <ed...@gmail.com> on 2016/04/14 05:29:07 UTC
Optimal indexing speed in Solr
Hi,
Would like to find out, what is the optimal indexing speed in Solr?
Previously, I managed to get more than 3GB/hour, but now the speed has drop
to 0.7GB/hr. What could be the potential reason behind this?
Besides the index size getting bigger, I have only added in more
collections into the core and added another field. Other than that nothing
else has been changed..
Could the source file which I'm indexing made a difference in the indexing
speed?
I'm using Solr 5.4.0 for now, but will be planning to migrate to Solr 6.0.0.
Regards,
Edwin
Re: Optimal indexing speed in Solr
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
Stupid phone autocorrect...
If you add updated documents of the same ID over time, optimizing your
collection(s) may help.
On Thu, Apr 14, 2016 at 7:50 AM, John Bickerstaff <jo...@johnbickerstaff.com>
wrote:
> If you delete a lot of documents over time, or if you add updated
> documents of the same I'd over time, optimizing your collection(s) may help.
> On Apr 14, 2016 3:52 AM, "Emir Arnautovic" <em...@sematext.com>
> wrote:
>
>> Hi Edwin,
>> Indexing speed depends on multiple factors: HW, Solr configurations and
>> load, documents, indexing client: More complex documents, more CPU time to
>> process each document before indexing structure is written down to disk.
>> Bigger the index, more heap is used, more frequent GCs. Maybe you are just
>> not sending enough doc to Solr to have such throughput.
>> The best way to pinpoint bottleneck is to use some monitoring tool. One
>> such tool is our SPM (http://sematext.com/spm) - it allows you to
>> monitor both Solr and OS metrics.
>>
>> HTH,
>> Emir
>>
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>>
>> On 14.04.2016 05:29, Zheng Lin Edwin Yeo wrote:
>>
>>> Hi,
>>>
>>> Would like to find out, what is the optimal indexing speed in Solr?
>>>
>>> Previously, I managed to get more than 3GB/hour, but now the speed has
>>> drop
>>> to 0.7GB/hr. What could be the potential reason behind this?
>>>
>>> Besides the index size getting bigger, I have only added in more
>>> collections into the core and added another field. Other than that
>>> nothing
>>> else has been changed..
>>>
>>> Could the source file which I'm indexing made a difference in the
>>> indexing
>>> speed?
>>>
>>> I'm using Solr 5.4.0 for now, but will be planning to migrate to Solr
>>> 6.0.0.
>>>
>>> Regards,
>>> Edwin
>>>
>>>
>>
Re: Optimal indexing speed in Solr
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
If you delete a lot of documents over time, or if you add updated documents
of the same I'd over time, optimizing your collection(s) may help.
On Apr 14, 2016 3:52 AM, "Emir Arnautovic" <em...@sematext.com>
wrote:
> Hi Edwin,
> Indexing speed depends on multiple factors: HW, Solr configurations and
> load, documents, indexing client: More complex documents, more CPU time to
> process each document before indexing structure is written down to disk.
> Bigger the index, more heap is used, more frequent GCs. Maybe you are just
> not sending enough doc to Solr to have such throughput.
> The best way to pinpoint bottleneck is to use some monitoring tool. One
> such tool is our SPM (http://sematext.com/spm) - it allows you to monitor
> both Solr and OS metrics.
>
> HTH,
> Emir
>
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On 14.04.2016 05:29, Zheng Lin Edwin Yeo wrote:
>
>> Hi,
>>
>> Would like to find out, what is the optimal indexing speed in Solr?
>>
>> Previously, I managed to get more than 3GB/hour, but now the speed has
>> drop
>> to 0.7GB/hr. What could be the potential reason behind this?
>>
>> Besides the index size getting bigger, I have only added in more
>> collections into the core and added another field. Other than that nothing
>> else has been changed..
>>
>> Could the source file which I'm indexing made a difference in the indexing
>> speed?
>>
>> I'm using Solr 5.4.0 for now, but will be planning to migrate to Solr
>> 6.0.0.
>>
>> Regards,
>> Edwin
>>
>>
>
Re: Optimal indexing speed in Solr
Posted by Emir Arnautovic <em...@sematext.com>.
Hi Edwin,
Indexing speed depends on multiple factors: HW, Solr configurations and
load, documents, indexing client: More complex documents, more CPU time
to process each document before indexing structure is written down to
disk. Bigger the index, more heap is used, more frequent GCs. Maybe you
are just not sending enough doc to Solr to have such throughput.
The best way to pinpoint bottleneck is to use some monitoring tool. One
such tool is our SPM (http://sematext.com/spm) - it allows you to
monitor both Solr and OS metrics.
HTH,
Emir
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On 14.04.2016 05:29, Zheng Lin Edwin Yeo wrote:
> Hi,
>
> Would like to find out, what is the optimal indexing speed in Solr?
>
> Previously, I managed to get more than 3GB/hour, but now the speed has drop
> to 0.7GB/hr. What could be the potential reason behind this?
>
> Besides the index size getting bigger, I have only added in more
> collections into the core and added another field. Other than that nothing
> else has been changed..
>
> Could the source file which I'm indexing made a difference in the indexing
> speed?
>
> I'm using Solr 5.4.0 for now, but will be planning to migrate to Solr 6.0.0.
>
> Regards,
> Edwin
>