You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mingfeng Yang <mf...@wisewindow.com> on 2013/02/14 20:46:41 UTC

fatest way to rebuild Solr index

I have a few Solr indexes, each with 20-200 millions documents, which were
indexed by querying multiple PostgreSQL databases.  If I do rebuild the
index by the same way, it would take a few months, because the PostgresSQL
query is slow.

Now, I need to do the following changes to all indexes.
1. delete a couple fields from the Solr index
2. add a couple new fields
3. change the type of one field from string to int

Luckily, all fields were "indexed and stored".   My plan is to query an old
index, and get values for all fields, and then add them into new index.

Any faster ways to build new indexes in my case?

Thanks,
Ming

Re: fatest way to rebuild Solr index

Posted by Mingfeng Yang <mf...@wisewindow.com>.
Shawn,

Awesome.  Exactly something I am looking for.

Thanks!
Ming


On Thu, Feb 14, 2013 at 12:00 PM, Shawn Heisey <so...@elyograg.org> wrote:

> On 2/14/2013 12:46 PM, Mingfeng Yang wrote:
>
>> I have a few Solr indexes, each with 20-200 millions documents, which were
>> indexed by querying multiple PostgreSQL databases.  If I do rebuild the
>> index by the same way, it would take a few months, because the PostgresSQL
>> query is slow.
>>
>> Now, I need to do the following changes to all indexes.
>> 1. delete a couple fields from the Solr index
>> 2. add a couple new fields
>> 3. change the type of one field from string to int
>>
>> Luckily, all fields were "indexed and stored".   My plan is to query an
>> old
>> index, and get values for all fields, and then add them into new index.
>>
>
> Using the DataImportHandler with SolrEntityProcessor is probably your best
> bet.  I believe you would want to avoid updating the source index while
> using this.
>
> http://wiki.apache.org/solr/**DataImportHandler#**SolrEntityProcessor<http://wiki.apache.org/solr/DataImportHandler#SolrEntityProcessor>
>
> Thanks,
> Shawn
>
>

Re: fatest way to rebuild Solr index

Posted by Shawn Heisey <so...@elyograg.org>.
On 2/14/2013 12:46 PM, Mingfeng Yang wrote:
> I have a few Solr indexes, each with 20-200 millions documents, which were
> indexed by querying multiple PostgreSQL databases.  If I do rebuild the
> index by the same way, it would take a few months, because the PostgresSQL
> query is slow.
>
> Now, I need to do the following changes to all indexes.
> 1. delete a couple fields from the Solr index
> 2. add a couple new fields
> 3. change the type of one field from string to int
>
> Luckily, all fields were "indexed and stored".   My plan is to query an old
> index, and get values for all fields, and then add them into new index.

Using the DataImportHandler with SolrEntityProcessor is probably your 
best bet.  I believe you would want to avoid updating the source index 
while using this.

http://wiki.apache.org/solr/DataImportHandler#SolrEntityProcessor

Thanks,
Shawn