You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Anuj Lal <al...@marinsoftware.com> on 2016/04/05 23:08:43 UTC
CompositId router
I am new to solr. Need some advice from more experienced solr team members
I am upgrading 4.4 solr cluster to 5.5
One of the step I am doing for upgrade is to bootstrap from existing 4.4 solr home ( after upgrading solr installation to 5.5)
All of the nodes comes up correctly and I can query data after upgrade. No issues here
I am facing two issues . In 4.4 my collection is using compositId router and routing field
After upgrade I am seeing cluster status router as implicit and collection router as implicit
I tried upgrading cluster status with shard range and router as composite and collection router as composite manually in zookeeper and restart cluster.
( Above I did based on this issue)
After this status I see correct cluster status and collection router.
When I update document, data is still not going to correct shard. Looks like it is still using implicit router.
Question
1. What is the correct step for migration composit routed collection to 5.5 .
2. If I create collection again with rest api, how can i migrate data from 4.4 to new collection
Appreciate any help
Thanks
Anuj
Re: CompositId router
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
I think that's how I would approach it. I used command-line instead of
rest api to create collection, but I think that just generates rest api
command via curl... so that will be no different as far as I can tell - I'm
just more comfortable on the command line.
Step 8 is the thing I'm not sure about - do you mean import using a tool
or do you mean simply copy the data directories?
It's worth a try, that's for sure - although this line in one of the URLs I
gave you gives me pause...
Solr 5 has no support for reading Lucene/Solr 3.x and earlier indexes. Be
sure to run the Lucene IndexUpgrader included with Solr 4.10 if you might
still have old 3x formatted segments in your index. Alternatively: fully
optimize your index with Solr 4.10 to make sure it consists only of one
up-to-date index segment.
What I would do is try it in a test environment and see if I run into
problems before worrying about everything that "could" go wrong...
I'm afraid I can't do you much good on the question of compositeId router -
I haven't dealt with that. I'm just hoping that a complete recreation of
the solr environment will resolve the problem...
On Wed, Apr 6, 2016 at 10:28 AM, Anuj Lal <al...@marinsoftware.com> wrote:
> Hi John , Shawn
>
> Thanks for replying to my query . Really appreciate your responses
>
> Ideally I’d like to do node by node rolling upgrade from 4.4 to 5.5
>
> But gave this approach of rolling upgrade because I faced issue with SolrJ
> 4.4 client connecting with 5.5 cluster or 5.5 solar client connecting with
> 4.4 cluster.
>
> I am fine with stopping whole cluster and upgrade all nodes to 5.5.
>
> What I don’t want to do recreate complete index for all clients from source
> system. At least I’d like to avoid recreate index from source as that will
> be really time consuming process for us.
>
> This is what I have done
> 1. Stop all nodes and zookeeper
> 2. Updated solr 4.4 to 5.5 on all cluster nodes
> 3. Deleted zookeeper data
> 4. Modify solr.xml, solrconfig.xml, created core.properties as per 5.5
> changes
> 5. Boot strap from existing solr home using this command
>
> ./solr-5.5.0/server/scripts/cloud-scripts/zkcli.sh -z <zookeepr server and
> port> -cmd bootstrap -solrhome /home/solr/solrcloud-shard1 -d <path to
> sole con directory> -n <cofig name> -c <collection name>
>
> 6. Start zookeeper
>
> 7. start all solr nodes
>
> all nodes comes up correctly. I can query data fro solr ui as well as from
> our application using solrJ client upgraded to 5.5 libs
>
> What I see that clusterstatus.json and /collections/<collectionName> shows
> implicit router not compositId router
>
> When I update document, it doesnt go to right shard but different shard
>
> Just for trying purpose, I manually updated clusterstatus.json and
> /collections/<CollectionName> with compositeId router and shards keys
> after stopping cluster and zookeeper and restar after update of json
> files
>
>
> Based on your reply, I think suggested steps are ( please confirm if these
> are appropriate)
>
>
> 1. Stop all nodes and zookeeper
> 2. Updated solr 4.4 to 5.5 on all cluster nodes
> 3. Deleted zookeeper data
> 4. Modify solr.xml, solrconfig.xml, created core.properties as per 5.5
> changes
> 5. start zookeeper
> 5. upload config to zookeeper
> 6. Create collection using rest api
> 7. start cluster
> 8. copy collection data from 4.5 to solr 5.5 data directory
>
>
> If you can share upgrade step/process document, that will be great
>
> Thanks
> Anuj
>
>
>
>
>
>
>
> On Apr 6, 2016, at 7:58 AM, John Bickerstaff <jo...@johnbickerstaff.com>
> wrote:
>
> I recently upgraded from 4.x to 5.5 -- it was a pain to figure it out, but
> it turns out to be fairly straightforward...
>
> Caveat: Because I run all my data into Kafka first, I was able to easily
> re-create my collections by running a microservice that pulls from Kafka
> and dumps into Solr.
>
> I have a rough draft document for how to "upgrade by replacement" in this
> way - basically, throw away the older Solr, build a new /chroot in
> Zookeeper and issue a few command line commands to upload configs (from the
> 4.x version) and create new collections in the 5.x version.
>
> If this is what you mean by "bootstrap" (Upgrade to 5.x but use your
> index/collection/core from a previous version) then perhaps I can help.
>
> There were instructions on the Solr wiki for the differences between 4.x
> and 5.x...
>
> But before I go into a lot more detail, please drop us a line and let me
> know if that's what you need...
>
> For reference you can look at this, but I have more refined instructions
> now:
>
>
> http://stackoverflow.com/questions/34909909/how-to-correctly-add-additional-solr-5-vm-nodes-to-solr-cloud?rq=1
>
> On Wed, Apr 6, 2016 at 7:01 AM, Shawn Heisey <ap...@elyograg.org> wrote:
>
> On 4/5/2016 3:08 PM, Anuj Lal wrote:
>
> I am new to solr. Need some advice from more experienced solr team
>
> members
>
>
> I am upgrading 4.4 solr cluster to 5.5
>
> One of the step I am doing for upgrade is to bootstrap from existing 4.4
>
> solr home ( after upgrading solr installation to 5.5)
>
> We'll need a lot more detail about *exactly* what "bootstrap" means
> here. The fact that you chose the word "bootstrap" to describe your
> process sets off a red flag in my mind, and might mean that you're doing
> this upgrade in a way that won't work like you're expecting it to.
>
> One particular question I'd like to have answered: Are you upgrading
> each server in place using the exact same zkHost string as the 4.4
> install was using, or are you trying to build up a new ZK
> chroot/database using the existing Solr home?
>
> I have not actually performed one of these SolrCloud upgrades, but I
> would expect that if you don't connect to the existing zookeeper
> database, it would probably behave exactly like you've described. If
> I'm completely wrong about how this should be done, somebody please let
> me know.
>
> Going from 4.4 to 5.5 is a significant upgrade (including one major
> release, 10 minor releases, and 10 bugfix releases) and you might find
> that the two versions won't coexist very well. The devs do try to make
> different versions compatible, but SolrCloud is evolving at an extremely
> rapid pace, so this cannot be guaranteed when the version difference is
> so large.
>
> Assuming I have a correct mental picture of how you got to where you
> are, there might be some things you could do to get it back on track,
> but that would require significant manual fiddling, and no guarantee of
> success.
>
> Thanks,
> Shawn
>
Re: CompositId router
Posted by Anuj Lal <al...@marinsoftware.com>.
Hi John , Shawn
Thanks for replying to my query . Really appreciate your responses
Ideally I’d like to do node by node rolling upgrade from 4.4 to 5.5
But gave this approach of rolling upgrade because I faced issue with SolrJ
4.4 client connecting with 5.5 cluster or 5.5 solar client connecting with
4.4 cluster.
I am fine with stopping whole cluster and upgrade all nodes to 5.5.
What I don’t want to do recreate complete index for all clients from source
system. At least I’d like to avoid recreate index from source as that will
be really time consuming process for us.
This is what I have done
1. Stop all nodes and zookeeper
2. Updated solr 4.4 to 5.5 on all cluster nodes
3. Deleted zookeeper data
4. Modify solr.xml, solrconfig.xml, created core.properties as per 5.5
changes
5. Boot strap from existing solr home using this command
./solr-5.5.0/server/scripts/cloud-scripts/zkcli.sh -z <zookeepr server and
port> -cmd bootstrap -solrhome /home/solr/solrcloud-shard1 -d <path to
sole con directory> -n <cofig name> -c <collection name>
6. Start zookeeper
7. start all solr nodes
all nodes comes up correctly. I can query data fro solr ui as well as from
our application using solrJ client upgraded to 5.5 libs
What I see that clusterstatus.json and /collections/<collectionName> shows
implicit router not compositId router
When I update document, it doesnt go to right shard but different shard
Just for trying purpose, I manually updated clusterstatus.json and
/collections/<CollectionName> with compositeId router and shards keys
after stopping cluster and zookeeper and restar after update of json files
Based on your reply, I think suggested steps are ( please confirm if these
are appropriate)
1. Stop all nodes and zookeeper
2. Updated solr 4.4 to 5.5 on all cluster nodes
3. Deleted zookeeper data
4. Modify solr.xml, solrconfig.xml, created core.properties as per 5.5
changes
5. start zookeeper
5. upload config to zookeeper
6. Create collection using rest api
7. start cluster
8. copy collection data from 4.5 to solr 5.5 data directory
If you can share upgrade step/process document, that will be great
Thanks
Anuj
On Apr 6, 2016, at 7:58 AM, John Bickerstaff <jo...@johnbickerstaff.com>
wrote:
I recently upgraded from 4.x to 5.5 -- it was a pain to figure it out, but
it turns out to be fairly straightforward...
Caveat: Because I run all my data into Kafka first, I was able to easily
re-create my collections by running a microservice that pulls from Kafka
and dumps into Solr.
I have a rough draft document for how to "upgrade by replacement" in this
way - basically, throw away the older Solr, build a new /chroot in
Zookeeper and issue a few command line commands to upload configs (from the
4.x version) and create new collections in the 5.x version.
If this is what you mean by "bootstrap" (Upgrade to 5.x but use your
index/collection/core from a previous version) then perhaps I can help.
There were instructions on the Solr wiki for the differences between 4.x
and 5.x...
But before I go into a lot more detail, please drop us a line and let me
know if that's what you need...
For reference you can look at this, but I have more refined instructions
now:
http://stackoverflow.com/questions/34909909/how-to-correctly-add-additional-solr-5-vm-nodes-to-solr-cloud?rq=1
On Wed, Apr 6, 2016 at 7:01 AM, Shawn Heisey <ap...@elyograg.org> wrote:
On 4/5/2016 3:08 PM, Anuj Lal wrote:
I am new to solr. Need some advice from more experienced solr team
members
I am upgrading 4.4 solr cluster to 5.5
One of the step I am doing for upgrade is to bootstrap from existing 4.4
solr home ( after upgrading solr installation to 5.5)
We'll need a lot more detail about *exactly* what "bootstrap" means
here. The fact that you chose the word "bootstrap" to describe your
process sets off a red flag in my mind, and might mean that you're doing
this upgrade in a way that won't work like you're expecting it to.
One particular question I'd like to have answered: Are you upgrading
each server in place using the exact same zkHost string as the 4.4
install was using, or are you trying to build up a new ZK
chroot/database using the existing Solr home?
I have not actually performed one of these SolrCloud upgrades, but I
would expect that if you don't connect to the existing zookeeper
database, it would probably behave exactly like you've described. If
I'm completely wrong about how this should be done, somebody please let
me know.
Going from 4.4 to 5.5 is a significant upgrade (including one major
release, 10 minor releases, and 10 bugfix releases) and you might find
that the two versions won't coexist very well. The devs do try to make
different versions compatible, but SolrCloud is evolving at an extremely
rapid pace, so this cannot be guaranteed when the version difference is
so large.
Assuming I have a correct mental picture of how you got to where you
are, there might be some things you could do to get it back on track,
but that would require significant manual fiddling, and no guarantee of
success.
Thanks,
Shawn
Re: CompositId router
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
I'll agree with Shawn too - munging Zookeeper by hand can lead to VERY
unexpected results...
My recommendation would be to start fresh with a new 5.x setup and a new
/chroot in Zookeeper.
(This can be deleted and recreated repeatedly if necessary - I know because
I did... a lot... before I got it right)
Then, pull your data out of your old core/collection either by using the
approach in the link I sent earlier or possibly copying over the core
directory and making the changes to the xml files necessary for 5.x
These links may help you decide how to proceed...
https://cwiki.apache.org/confluence/display/solr/Upgrading+Solr
https://cwiki.apache.org/confluence/display/solr/Upgrading+a+Solr+4.x+Cluster+to+Solr+5.0
https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+4+to+Solr+5
On Wed, Apr 6, 2016 at 8:58 AM, John Bickerstaff <jo...@johnbickerstaff.com>
wrote:
> I recently upgraded from 4.x to 5.5 -- it was a pain to figure it out, but
> it turns out to be fairly straightforward...
>
> Caveat: Because I run all my data into Kafka first, I was able to easily
> re-create my collections by running a microservice that pulls from Kafka
> and dumps into Solr.
>
> I have a rough draft document for how to "upgrade by replacement" in this
> way - basically, throw away the older Solr, build a new /chroot in
> Zookeeper and issue a few command line commands to upload configs (from the
> 4.x version) and create new collections in the 5.x version.
>
> If this is what you mean by "bootstrap" (Upgrade to 5.x but use your
> index/collection/core from a previous version) then perhaps I can help.
>
> There were instructions on the Solr wiki for the differences between 4.x
> and 5.x...
>
> But before I go into a lot more detail, please drop us a line and let me
> know if that's what you need...
>
> For reference you can look at this, but I have more refined instructions
> now:
>
>
> http://stackoverflow.com/questions/34909909/how-to-correctly-add-additional-solr-5-vm-nodes-to-solr-cloud?rq=1
>
> On Wed, Apr 6, 2016 at 7:01 AM, Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 4/5/2016 3:08 PM, Anuj Lal wrote:
>> > I am new to solr. Need some advice from more experienced solr team
>> members
>> >
>> > I am upgrading 4.4 solr cluster to 5.5
>> >
>> > One of the step I am doing for upgrade is to bootstrap from existing
>> 4.4 solr home ( after upgrading solr installation to 5.5)
>>
>> We'll need a lot more detail about *exactly* what "bootstrap" means
>> here. The fact that you chose the word "bootstrap" to describe your
>> process sets off a red flag in my mind, and might mean that you're doing
>> this upgrade in a way that won't work like you're expecting it to.
>>
>> One particular question I'd like to have answered: Are you upgrading
>> each server in place using the exact same zkHost string as the 4.4
>> install was using, or are you trying to build up a new ZK
>> chroot/database using the existing Solr home?
>>
>> I have not actually performed one of these SolrCloud upgrades, but I
>> would expect that if you don't connect to the existing zookeeper
>> database, it would probably behave exactly like you've described. If
>> I'm completely wrong about how this should be done, somebody please let
>> me know.
>>
>> Going from 4.4 to 5.5 is a significant upgrade (including one major
>> release, 10 minor releases, and 10 bugfix releases) and you might find
>> that the two versions won't coexist very well. The devs do try to make
>> different versions compatible, but SolrCloud is evolving at an extremely
>> rapid pace, so this cannot be guaranteed when the version difference is
>> so large.
>>
>> Assuming I have a correct mental picture of how you got to where you
>> are, there might be some things you could do to get it back on track,
>> but that would require significant manual fiddling, and no guarantee of
>> success.
>>
>> Thanks,
>> Shawn
>>
>>
>
Re: CompositId router
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
I recently upgraded from 4.x to 5.5 -- it was a pain to figure it out, but
it turns out to be fairly straightforward...
Caveat: Because I run all my data into Kafka first, I was able to easily
re-create my collections by running a microservice that pulls from Kafka
and dumps into Solr.
I have a rough draft document for how to "upgrade by replacement" in this
way - basically, throw away the older Solr, build a new /chroot in
Zookeeper and issue a few command line commands to upload configs (from the
4.x version) and create new collections in the 5.x version.
If this is what you mean by "bootstrap" (Upgrade to 5.x but use your
index/collection/core from a previous version) then perhaps I can help.
There were instructions on the Solr wiki for the differences between 4.x
and 5.x...
But before I go into a lot more detail, please drop us a line and let me
know if that's what you need...
For reference you can look at this, but I have more refined instructions
now:
http://stackoverflow.com/questions/34909909/how-to-correctly-add-additional-solr-5-vm-nodes-to-solr-cloud?rq=1
On Wed, Apr 6, 2016 at 7:01 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 4/5/2016 3:08 PM, Anuj Lal wrote:
> > I am new to solr. Need some advice from more experienced solr team
> members
> >
> > I am upgrading 4.4 solr cluster to 5.5
> >
> > One of the step I am doing for upgrade is to bootstrap from existing 4.4
> solr home ( after upgrading solr installation to 5.5)
>
> We'll need a lot more detail about *exactly* what "bootstrap" means
> here. The fact that you chose the word "bootstrap" to describe your
> process sets off a red flag in my mind, and might mean that you're doing
> this upgrade in a way that won't work like you're expecting it to.
>
> One particular question I'd like to have answered: Are you upgrading
> each server in place using the exact same zkHost string as the 4.4
> install was using, or are you trying to build up a new ZK
> chroot/database using the existing Solr home?
>
> I have not actually performed one of these SolrCloud upgrades, but I
> would expect that if you don't connect to the existing zookeeper
> database, it would probably behave exactly like you've described. If
> I'm completely wrong about how this should be done, somebody please let
> me know.
>
> Going from 4.4 to 5.5 is a significant upgrade (including one major
> release, 10 minor releases, and 10 bugfix releases) and you might find
> that the two versions won't coexist very well. The devs do try to make
> different versions compatible, but SolrCloud is evolving at an extremely
> rapid pace, so this cannot be guaranteed when the version difference is
> so large.
>
> Assuming I have a correct mental picture of how you got to where you
> are, there might be some things you could do to get it back on track,
> but that would require significant manual fiddling, and no guarantee of
> success.
>
> Thanks,
> Shawn
>
>
Re: CompositId router
Posted by Shawn Heisey <ap...@elyograg.org>.
On 4/5/2016 3:08 PM, Anuj Lal wrote:
> I am new to solr. Need some advice from more experienced solr team members
>
> I am upgrading 4.4 solr cluster to 5.5
>
> One of the step I am doing for upgrade is to bootstrap from existing 4.4 solr home ( after upgrading solr installation to 5.5)
We'll need a lot more detail about *exactly* what "bootstrap" means
here. The fact that you chose the word "bootstrap" to describe your
process sets off a red flag in my mind, and might mean that you're doing
this upgrade in a way that won't work like you're expecting it to.
One particular question I'd like to have answered: Are you upgrading
each server in place using the exact same zkHost string as the 4.4
install was using, or are you trying to build up a new ZK
chroot/database using the existing Solr home?
I have not actually performed one of these SolrCloud upgrades, but I
would expect that if you don't connect to the existing zookeeper
database, it would probably behave exactly like you've described. If
I'm completely wrong about how this should be done, somebody please let
me know.
Going from 4.4 to 5.5 is a significant upgrade (including one major
release, 10 minor releases, and 10 bugfix releases) and you might find
that the two versions won't coexist very well. The devs do try to make
different versions compatible, but SolrCloud is evolving at an extremely
rapid pace, so this cannot be guaranteed when the version difference is
so large.
Assuming I have a correct mental picture of how you got to where you
are, there might be some things you could do to get it back on track,
but that would require significant manual fiddling, and no guarantee of
success.
Thanks,
Shawn
Re: CompositId router
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
In terms of #2, this might be of use...
https://wiki.apache.org/solr/HowToReindex
On Tue, Apr 5, 2016 at 3:08 PM, Anuj Lal <al...@marinsoftware.com> wrote:
> I am new to solr. Need some advice from more experienced solr team
> members
>
> I am upgrading 4.4 solr cluster to 5.5
>
>
> One of the step I am doing for upgrade is to bootstrap from existing 4.4
> solr home ( after upgrading solr installation to 5.5)
>
> All of the nodes comes up correctly and I can query data after upgrade. No
> issues here
>
> I am facing two issues . In 4.4 my collection is using compositId router
> and routing field
>
> After upgrade I am seeing cluster status router as implicit and collection
> router as implicit
>
> I tried upgrading cluster status with shard range and router as composite
> and collection router as composite manually in zookeeper and restart
> cluster.
> ( Above I did based on this issue)
>
> After this status I see correct cluster status and collection router.
>
> When I update document, data is still not going to correct shard. Looks
> like it is still using implicit router.
>
> Question
> 1. What is the correct step for migration composit routed collection to
> 5.5 .
> 2. If I create collection again with rest api, how can i migrate data from
> 4.4 to new collection
>
> Appreciate any help
>
> Thanks
> Anuj
>
>
>
>