You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Krizelle Mae Hernandez <kr...@sas.com> on 2018/09/26 06:36:37 UTC
SOLR Index Time Running Optimization
Hi.
Our SOLR currently is running approximately 39hours for Full and Delta Import. I would like to ask for your assistance on how can we shorten the 39hours run time in any possible solution?
For SOLR version, we are using solr 5.3.1.
Regards,
Krizelle Mae M. Hernandez
Re: SOLR Index Time Running Optimization
Posted by Walter Underwood <wu...@wunderwood.org>.
How long does the query take when it is run directly, without Solr?
For our DIH queries, Solr was not the slow part. It took 90 minutes
directly or with DIH. With our big cluster, I’ve seen indexing rates of
one million docs per minute.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Sep 26, 2018, at 9:44 AM, Jan Høydahl <ja...@cominvent.com> wrote:
>
> With DIH you are doing indexing single-threaded. You should be able to configure multiple DIH's on the same collection and then partition the data between them, issuing slightly different SQL to each. But I don't exactly know what that would look like.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>> 26. sep. 2018 kl. 14:30 skrev Susheel Kumar <su...@gmail.com>:
>>
>> Also are you using Solr data import? That will be much slower compare to if
>> you write our own little indexer which does indexing in batches and with
>> multiple threads.
>>
>> On Wed, Sep 26, 2018 at 8:00 AM Vincenzo D'Amore <v....@gmail.com> wrote:
>>
>>> Hi, I know this is the shortest way but, had you tried to add more core or
>>> CPU to your solr instances? How big is you collection in terms of GB and
>>> number of documents?
>>>
>>> Ciao,
>>> Vincenzo
>>>
>>>
>>>> On 26 Sep 2018, at 08:36, Krizelle Mae Hernandez <
>>> krizellemae.martinez@sas.com> wrote:
>>>>
>>>> Hi.
>>>>
>>>> Our SOLR currently is running approximately 39hours for Full and Delta
>>> Import. I would like to ask for your assistance on how can we shorten the
>>> 39hours run time in any possible solution?
>>>> For SOLR version, we are using solr 5.3.1.
>>>>
>>>> Regards,
>>>> Krizelle Mae M. Hernandez
>>>
>
Re: SOLR Index Time Running Optimization
Posted by Jan Høydahl <ja...@cominvent.com>.
With DIH you are doing indexing single-threaded. You should be able to configure multiple DIH's on the same collection and then partition the data between them, issuing slightly different SQL to each. But I don't exactly know what that would look like.
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
> 26. sep. 2018 kl. 14:30 skrev Susheel Kumar <su...@gmail.com>:
>
> Also are you using Solr data import? That will be much slower compare to if
> you write our own little indexer which does indexing in batches and with
> multiple threads.
>
> On Wed, Sep 26, 2018 at 8:00 AM Vincenzo D'Amore <v....@gmail.com> wrote:
>
>> Hi, I know this is the shortest way but, had you tried to add more core or
>> CPU to your solr instances? How big is you collection in terms of GB and
>> number of documents?
>>
>> Ciao,
>> Vincenzo
>>
>>
>>> On 26 Sep 2018, at 08:36, Krizelle Mae Hernandez <
>> krizellemae.martinez@sas.com> wrote:
>>>
>>> Hi.
>>>
>>> Our SOLR currently is running approximately 39hours for Full and Delta
>> Import. I would like to ask for your assistance on how can we shorten the
>> 39hours run time in any possible solution?
>>> For SOLR version, we are using solr 5.3.1.
>>>
>>> Regards,
>>> Krizelle Mae M. Hernandez
>>
Re: SOLR Index Time Running Optimization
Posted by Susheel Kumar <su...@gmail.com>.
Also are you using Solr data import? That will be much slower compare to if
you write our own little indexer which does indexing in batches and with
multiple threads.
On Wed, Sep 26, 2018 at 8:00 AM Vincenzo D'Amore <v....@gmail.com> wrote:
> Hi, I know this is the shortest way but, had you tried to add more core or
> CPU to your solr instances? How big is you collection in terms of GB and
> number of documents?
>
> Ciao,
> Vincenzo
>
>
> > On 26 Sep 2018, at 08:36, Krizelle Mae Hernandez <
> krizellemae.martinez@sas.com> wrote:
> >
> > Hi.
> >
> > Our SOLR currently is running approximately 39hours for Full and Delta
> Import. I would like to ask for your assistance on how can we shorten the
> 39hours run time in any possible solution?
> > For SOLR version, we are using solr 5.3.1.
> >
> > Regards,
> > Krizelle Mae M. Hernandez
>
Re: SOLR Index Time Running Optimization
Posted by Vincenzo D'Amore <v....@gmail.com>.
Hi, I know this is the shortest way but, had you tried to add more core or CPU to your solr instances? How big is you collection in terms of GB and number of documents?
Ciao,
Vincenzo
> On 26 Sep 2018, at 08:36, Krizelle Mae Hernandez <kr...@sas.com> wrote:
>
> Hi.
>
> Our SOLR currently is running approximately 39hours for Full and Delta Import. I would like to ask for your assistance on how can we shorten the 39hours run time in any possible solution?
> For SOLR version, we are using solr 5.3.1.
>
> Regards,
> Krizelle Mae M. Hernandez