You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Neeraj Bhatt <ne...@gmail.com> on 2016/10/04 03:01:18 UTC
Upgrading from Solr cloud 4.1 to 6.2
Hello All
We are trying to upgrade our production solr with 10 million documents from
solr cloud (5 shards, 5 nodes, one collection, 3 replica) 4.1 to 6.2
How to upgrade the lucene index created by solr. Should I go into indexes
created by each shard and upgrade and replicate it manually ? Also I tried
using Index upgrader in one replica of one shard as a test but it gives
error as it is looking for _4c.si file and it is not there
Any idea what is the easy way to upgrade solr cloud with a 10m repsoitory
Thanks
neeraj
Re: Upgrading from Solr cloud 4.1 to 6.2
Posted by Erick Erickson <er...@gmail.com>.
As both Jan and I mentioned, reindexing is _always_ the least error prone,
so please do that if at all possible.... And while you're at it it's a fine time
to make any tweaks like adding DocValues for fields you sort or
facet on, perhaps enable the new return stored fields from doc values
functionality and the like.
However, if you absolutely _have_ to upgrade here's what I'd do.
1> create a leader-only, 5-node 6x cluster
2> upgrade one index from each shard 4x->6x and put it on the
correct shard on your new cluster. This is perhaps the most
error-prone. Did you get the index with the same hash range on
the right node for each shard?
3> verify that it's consistent.
4> Use the collections API to ADDREPLICA on the new system
to build it out to 3 replicas. That'll automatically replicate the index
from the leader. You can choose exactly what node each replica
goes on, although Solr will do a pretty good job of spreading
them out and you can even have those decisions done by a set
of rules you specify...
Best,
Erick
On Fri, Oct 7, 2016 at 4:08 AM, Neeraj Bhatt <ne...@gmail.com> wrote:
> Thanks Jan for clarifying, I think I will pull all documents from data
> source again as you and Eric suggested
>
> Thanks
> neeraj
>
> On Fri, Oct 7, 2016 at 2:38 PM, Jan Høydahl <ja...@cominvent.com> wrote:
>
>> As Erick suggests, you should setup an empty 6.x environment,
>> create an empty collection with shards=5 replicationFactor=3
>> and then re-index all your content from your data source. Once that
>> is in, you can decommission your old cluster.
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>> > 7. okt. 2016 kl. 09.55 skrev Neeraj Bhatt <ne...@gmail.com>:
>> >
>> > Hi Eric
>> >
>> > Thanks for suggestion I was able to upgrade one shard one replica data
>> from
>> > 4.1 to 6.2 through index upgrader, but the new problem is since we were
>> > using solr cloud with multiple shards(5) with some replication (3) so do
>> I
>> > need to manually copy all index directory data for each shard upgrade and
>> > for each replica and place it in new index directory of solr 6.2 ?
>> >
>> > This seems to be error prone as there will be 15 index directories
>> >
>> > thanks
>> >
>> > On Tue, Oct 4, 2016 at 8:35 AM, Erick Erickson <er...@gmail.com>
>> > wrote:
>> >
>> >> the very easiest way is to re-index. 10M documents shouldn't take
>> >> very long unless they're no longer available...
>> >>
>> >> When you say you tried to use the index upgrader, which one? You'd
>> >> have to use the one distributed with 5.x to upgrade from 4.x->5.x, then
>> >> use the one distributed with 6x to go from 5.x->6.x.
>> >>
>> >>
>> >> Best,
>> >> Erick
>> >>
>> >> On Mon, Oct 3, 2016 at 8:01 PM, Neeraj Bhatt <neerajbhatt2000@gmail.com
>> >
>> >> wrote:
>> >>> Hello All
>> >>>
>> >>> We are trying to upgrade our production solr with 10 million documents
>> >> from
>> >>> solr cloud (5 shards, 5 nodes, one collection, 3 replica) 4.1 to 6.2
>> >>>
>> >>> How to upgrade the lucene index created by solr. Should I go into
>> indexes
>> >>> created by each shard and upgrade and replicate it manually ? Also I
>> >> tried
>> >>> using Index upgrader in one replica of one shard as a test but it gives
>> >>> error as it is looking for _4c.si file and it is not there
>> >>>
>> >>> Any idea what is the easy way to upgrade solr cloud with a 10m
>> repsoitory
>> >>>
>> >>> Thanks
>> >>> neeraj
>> >>
>>
>>
Re: Upgrading from Solr cloud 4.1 to 6.2
Posted by Neeraj Bhatt <ne...@gmail.com>.
Thanks Jan for clarifying, I think I will pull all documents from data
source again as you and Eric suggested
Thanks
neeraj
On Fri, Oct 7, 2016 at 2:38 PM, Jan Høydahl <ja...@cominvent.com> wrote:
> As Erick suggests, you should setup an empty 6.x environment,
> create an empty collection with shards=5 replicationFactor=3
> and then re-index all your content from your data source. Once that
> is in, you can decommission your old cluster.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 7. okt. 2016 kl. 09.55 skrev Neeraj Bhatt <ne...@gmail.com>:
> >
> > Hi Eric
> >
> > Thanks for suggestion I was able to upgrade one shard one replica data
> from
> > 4.1 to 6.2 through index upgrader, but the new problem is since we were
> > using solr cloud with multiple shards(5) with some replication (3) so do
> I
> > need to manually copy all index directory data for each shard upgrade and
> > for each replica and place it in new index directory of solr 6.2 ?
> >
> > This seems to be error prone as there will be 15 index directories
> >
> > thanks
> >
> > On Tue, Oct 4, 2016 at 8:35 AM, Erick Erickson <er...@gmail.com>
> > wrote:
> >
> >> the very easiest way is to re-index. 10M documents shouldn't take
> >> very long unless they're no longer available...
> >>
> >> When you say you tried to use the index upgrader, which one? You'd
> >> have to use the one distributed with 5.x to upgrade from 4.x->5.x, then
> >> use the one distributed with 6x to go from 5.x->6.x.
> >>
> >>
> >> Best,
> >> Erick
> >>
> >> On Mon, Oct 3, 2016 at 8:01 PM, Neeraj Bhatt <neerajbhatt2000@gmail.com
> >
> >> wrote:
> >>> Hello All
> >>>
> >>> We are trying to upgrade our production solr with 10 million documents
> >> from
> >>> solr cloud (5 shards, 5 nodes, one collection, 3 replica) 4.1 to 6.2
> >>>
> >>> How to upgrade the lucene index created by solr. Should I go into
> indexes
> >>> created by each shard and upgrade and replicate it manually ? Also I
> >> tried
> >>> using Index upgrader in one replica of one shard as a test but it gives
> >>> error as it is looking for _4c.si file and it is not there
> >>>
> >>> Any idea what is the easy way to upgrade solr cloud with a 10m
> repsoitory
> >>>
> >>> Thanks
> >>> neeraj
> >>
>
>
Re: Upgrading from Solr cloud 4.1 to 6.2
Posted by Jan Høydahl <ja...@cominvent.com>.
As Erick suggests, you should setup an empty 6.x environment,
create an empty collection with shards=5 replicationFactor=3
and then re-index all your content from your data source. Once that
is in, you can decommission your old cluster.
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
> 7. okt. 2016 kl. 09.55 skrev Neeraj Bhatt <ne...@gmail.com>:
>
> Hi Eric
>
> Thanks for suggestion I was able to upgrade one shard one replica data from
> 4.1 to 6.2 through index upgrader, but the new problem is since we were
> using solr cloud with multiple shards(5) with some replication (3) so do I
> need to manually copy all index directory data for each shard upgrade and
> for each replica and place it in new index directory of solr 6.2 ?
>
> This seems to be error prone as there will be 15 index directories
>
> thanks
>
> On Tue, Oct 4, 2016 at 8:35 AM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> the very easiest way is to re-index. 10M documents shouldn't take
>> very long unless they're no longer available...
>>
>> When you say you tried to use the index upgrader, which one? You'd
>> have to use the one distributed with 5.x to upgrade from 4.x->5.x, then
>> use the one distributed with 6x to go from 5.x->6.x.
>>
>>
>> Best,
>> Erick
>>
>> On Mon, Oct 3, 2016 at 8:01 PM, Neeraj Bhatt <ne...@gmail.com>
>> wrote:
>>> Hello All
>>>
>>> We are trying to upgrade our production solr with 10 million documents
>> from
>>> solr cloud (5 shards, 5 nodes, one collection, 3 replica) 4.1 to 6.2
>>>
>>> How to upgrade the lucene index created by solr. Should I go into indexes
>>> created by each shard and upgrade and replicate it manually ? Also I
>> tried
>>> using Index upgrader in one replica of one shard as a test but it gives
>>> error as it is looking for _4c.si file and it is not there
>>>
>>> Any idea what is the easy way to upgrade solr cloud with a 10m repsoitory
>>>
>>> Thanks
>>> neeraj
>>
Re: Upgrading from Solr cloud 4.1 to 6.2
Posted by Neeraj Bhatt <ne...@gmail.com>.
Hi Eric
Thanks for suggestion I was able to upgrade one shard one replica data from
4.1 to 6.2 through index upgrader, but the new problem is since we were
using solr cloud with multiple shards(5) with some replication (3) so do I
need to manually copy all index directory data for each shard upgrade and
for each replica and place it in new index directory of solr 6.2 ?
This seems to be error prone as there will be 15 index directories
thanks
On Tue, Oct 4, 2016 at 8:35 AM, Erick Erickson <er...@gmail.com>
wrote:
> the very easiest way is to re-index. 10M documents shouldn't take
> very long unless they're no longer available...
>
> When you say you tried to use the index upgrader, which one? You'd
> have to use the one distributed with 5.x to upgrade from 4.x->5.x, then
> use the one distributed with 6x to go from 5.x->6.x.
>
>
> Best,
> Erick
>
> On Mon, Oct 3, 2016 at 8:01 PM, Neeraj Bhatt <ne...@gmail.com>
> wrote:
> > Hello All
> >
> > We are trying to upgrade our production solr with 10 million documents
> from
> > solr cloud (5 shards, 5 nodes, one collection, 3 replica) 4.1 to 6.2
> >
> > How to upgrade the lucene index created by solr. Should I go into indexes
> > created by each shard and upgrade and replicate it manually ? Also I
> tried
> > using Index upgrader in one replica of one shard as a test but it gives
> > error as it is looking for _4c.si file and it is not there
> >
> > Any idea what is the easy way to upgrade solr cloud with a 10m repsoitory
> >
> > Thanks
> > neeraj
>
Re: Upgrading from Solr cloud 4.1 to 6.2
Posted by Erick Erickson <er...@gmail.com>.
the very easiest way is to re-index. 10M documents shouldn't take
very long unless they're no longer available...
When you say you tried to use the index upgrader, which one? You'd
have to use the one distributed with 5.x to upgrade from 4.x->5.x, then
use the one distributed with 6x to go from 5.x->6.x.
Best,
Erick
On Mon, Oct 3, 2016 at 8:01 PM, Neeraj Bhatt <ne...@gmail.com> wrote:
> Hello All
>
> We are trying to upgrade our production solr with 10 million documents from
> solr cloud (5 shards, 5 nodes, one collection, 3 replica) 4.1 to 6.2
>
> How to upgrade the lucene index created by solr. Should I go into indexes
> created by each shard and upgrade and replicate it manually ? Also I tried
> using Index upgrader in one replica of one shard as a test but it gives
> error as it is looking for _4c.si file and it is not there
>
> Any idea what is the easy way to upgrade solr cloud with a 10m repsoitory
>
> Thanks
> neeraj