You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Otis Gospodnetic <ot...@gmail.com> on 2013/07/25 13:09:29 UTC
Performance vs. maxBufferedAddsPerServer=10
Hi,
Context:
* https://issues.apache.org/jira/browse/SOLR-4956
* http://search-lucene.com/c/Solr:/core/src/java/org/apache/solr/update/SolrCmdDistributor.java%7C%7CmaxBufferedAddsPerServer
As you can see, maxBufferedAddsPerServer = 10.
We have an app that sends 20K docs to SolrCloud using CloudSolrServer.
We batch 20K docs for performance reasons. But then the receiving node
ends up sending VERY small batches of just 10 docs around for indexing
and we lose the benefit of batching those 20K docs in the first place.
Our app is "add only".
Is there anything one can do to avoid performance loss associated with
maxBufferedAddsPerServer=10?
Thanks,
Otis
--
Solr & ElasticSearch Support -- http://sematext.com/
Performance Monitoring -- http://sematext.com/spm
Re: Performance vs. maxBufferedAddsPerServer=10
Posted by Mark Miller <ma...@gmail.com>.
Yes, the internal document forwarding path is different and does not use the CloudSolrServer. It currently works with a buffer of 10.
- Mark
On Jul 29, 2013, at 3:10 PM, Erick Erickson <er...@gmail.com> wrote:
> Why wouldn't it? Or are you saying that the routing to replicas
> from the leader also 10/packet? Hmmm, hadn't thought of that...
>
> On Mon, Jul 29, 2013 at 7:58 AM, Mark Miller <ma...@gmail.com> wrote:
>> SOLR-4816 won't address this - it will just speed up *different* parts. There are other things that will need to be done to speed up that part.
>>
>> - Mark
>>
>> On Jul 26, 2013, at 3:53 PM, Erick Erickson <er...@gmail.com> wrote:
>>
>>> This is current a hard-coded limit from what I've understood. From what
>>> I remember, Mark said Yonik said that there are reasons to make the
>>> packets that size. But whether this is empirically a Good Thing I don't know.
>>>
>>> SOLR-4816 will address this a different way by making SolrJ batch up
>>> the docs and send them to the right leader, which should pretty much remove
>>> any performance consideration here.
>>>
>>> There's some anecdotal evidence that changing that in the code might
>>> improve throughput, but I don't remember the details.
>>>
>>> FWIW
>>> Erick
>>>
>>> On Thu, Jul 25, 2013 at 7:09 AM, Otis Gospodnetic
>>> <ot...@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> Context:
>>>> * https://issues.apache.org/jira/browse/SOLR-4956
>>>> * http://search-lucene.com/c/Solr:/core/src/java/org/apache/solr/update/SolrCmdDistributor.java%7C%7CmaxBufferedAddsPerServer
>>>>
>>>> As you can see, maxBufferedAddsPerServer = 10.
>>>>
>>>> We have an app that sends 20K docs to SolrCloud using CloudSolrServer.
>>>> We batch 20K docs for performance reasons. But then the receiving node
>>>> ends up sending VERY small batches of just 10 docs around for indexing
>>>> and we lose the benefit of batching those 20K docs in the first place.
>>>>
>>>> Our app is "add only".
>>>>
>>>> Is there anything one can do to avoid performance loss associated with
>>>> maxBufferedAddsPerServer=10?
>>>>
>>>> Thanks,
>>>> Otis
>>>> --
>>>> Solr & ElasticSearch Support -- http://sematext.com/
>>>> Performance Monitoring -- http://sematext.com/spm
>>
Re: Performance vs. maxBufferedAddsPerServer=10
Posted by Erick Erickson <er...@gmail.com>.
Why wouldn't it? Or are you saying that the routing to replicas
from the leader also 10/packet? Hmmm, hadn't thought of that...
On Mon, Jul 29, 2013 at 7:58 AM, Mark Miller <ma...@gmail.com> wrote:
> SOLR-4816 won't address this - it will just speed up *different* parts. There are other things that will need to be done to speed up that part.
>
> - Mark
>
> On Jul 26, 2013, at 3:53 PM, Erick Erickson <er...@gmail.com> wrote:
>
>> This is current a hard-coded limit from what I've understood. From what
>> I remember, Mark said Yonik said that there are reasons to make the
>> packets that size. But whether this is empirically a Good Thing I don't know.
>>
>> SOLR-4816 will address this a different way by making SolrJ batch up
>> the docs and send them to the right leader, which should pretty much remove
>> any performance consideration here.
>>
>> There's some anecdotal evidence that changing that in the code might
>> improve throughput, but I don't remember the details.
>>
>> FWIW
>> Erick
>>
>> On Thu, Jul 25, 2013 at 7:09 AM, Otis Gospodnetic
>> <ot...@gmail.com> wrote:
>>> Hi,
>>>
>>> Context:
>>> * https://issues.apache.org/jira/browse/SOLR-4956
>>> * http://search-lucene.com/c/Solr:/core/src/java/org/apache/solr/update/SolrCmdDistributor.java%7C%7CmaxBufferedAddsPerServer
>>>
>>> As you can see, maxBufferedAddsPerServer = 10.
>>>
>>> We have an app that sends 20K docs to SolrCloud using CloudSolrServer.
>>> We batch 20K docs for performance reasons. But then the receiving node
>>> ends up sending VERY small batches of just 10 docs around for indexing
>>> and we lose the benefit of batching those 20K docs in the first place.
>>>
>>> Our app is "add only".
>>>
>>> Is there anything one can do to avoid performance loss associated with
>>> maxBufferedAddsPerServer=10?
>>>
>>> Thanks,
>>> Otis
>>> --
>>> Solr & ElasticSearch Support -- http://sematext.com/
>>> Performance Monitoring -- http://sematext.com/spm
>
Re: Performance vs. maxBufferedAddsPerServer=10
Posted by Mark Miller <ma...@gmail.com>.
SOLR-4816 won't address this - it will just speed up *different* parts. There are other things that will need to be done to speed up that part.
- Mark
On Jul 26, 2013, at 3:53 PM, Erick Erickson <er...@gmail.com> wrote:
> This is current a hard-coded limit from what I've understood. From what
> I remember, Mark said Yonik said that there are reasons to make the
> packets that size. But whether this is empirically a Good Thing I don't know.
>
> SOLR-4816 will address this a different way by making SolrJ batch up
> the docs and send them to the right leader, which should pretty much remove
> any performance consideration here.
>
> There's some anecdotal evidence that changing that in the code might
> improve throughput, but I don't remember the details.
>
> FWIW
> Erick
>
> On Thu, Jul 25, 2013 at 7:09 AM, Otis Gospodnetic
> <ot...@gmail.com> wrote:
>> Hi,
>>
>> Context:
>> * https://issues.apache.org/jira/browse/SOLR-4956
>> * http://search-lucene.com/c/Solr:/core/src/java/org/apache/solr/update/SolrCmdDistributor.java%7C%7CmaxBufferedAddsPerServer
>>
>> As you can see, maxBufferedAddsPerServer = 10.
>>
>> We have an app that sends 20K docs to SolrCloud using CloudSolrServer.
>> We batch 20K docs for performance reasons. But then the receiving node
>> ends up sending VERY small batches of just 10 docs around for indexing
>> and we lose the benefit of batching those 20K docs in the first place.
>>
>> Our app is "add only".
>>
>> Is there anything one can do to avoid performance loss associated with
>> maxBufferedAddsPerServer=10?
>>
>> Thanks,
>> Otis
>> --
>> Solr & ElasticSearch Support -- http://sematext.com/
>> Performance Monitoring -- http://sematext.com/spm
Re: Performance vs. maxBufferedAddsPerServer=10
Posted by Erick Erickson <er...@gmail.com>.
This is current a hard-coded limit from what I've understood. From what
I remember, Mark said Yonik said that there are reasons to make the
packets that size. But whether this is empirically a Good Thing I don't know.
SOLR-4816 will address this a different way by making SolrJ batch up
the docs and send them to the right leader, which should pretty much remove
any performance consideration here.
There's some anecdotal evidence that changing that in the code might
improve throughput, but I don't remember the details.
FWIW
Erick
On Thu, Jul 25, 2013 at 7:09 AM, Otis Gospodnetic
<ot...@gmail.com> wrote:
> Hi,
>
> Context:
> * https://issues.apache.org/jira/browse/SOLR-4956
> * http://search-lucene.com/c/Solr:/core/src/java/org/apache/solr/update/SolrCmdDistributor.java%7C%7CmaxBufferedAddsPerServer
>
> As you can see, maxBufferedAddsPerServer = 10.
>
> We have an app that sends 20K docs to SolrCloud using CloudSolrServer.
> We batch 20K docs for performance reasons. But then the receiving node
> ends up sending VERY small batches of just 10 docs around for indexing
> and we lose the benefit of batching those 20K docs in the first place.
>
> Our app is "add only".
>
> Is there anything one can do to avoid performance loss associated with
> maxBufferedAddsPerServer=10?
>
> Thanks,
> Otis
> --
> Solr & ElasticSearch Support -- http://sematext.com/
> Performance Monitoring -- http://sematext.com/spm