Solrcloud v7 -> v8 upgrade question

[Richard <so...@okcoder.com>]

Hi folks!

I run a solrcloud deployment and we're about to start the v7->v8 
upgrade. It's large enough that building fully parallel infra isn't 
palatable so we were looking at mixed / in place upgrades.

 From my initial reading I have a few questions I was hoping y'all could 
help with:

1. The upgrade process[1] suggests an N+/-1 compat story, e.g. 8.11.2 
should be able to co-mingle with 7.>3 but I haven't turned up any firm 
statements about that.

2. Is there forward/backward compatibility between segment/index files 
between version? e.g. we have automatic core recovery from backups in 
the event that a node fails. If I'm running a v8 node that fails after a 
backup has been taken and the core gets restored to a v7 does that sound 
problematic? (I'm guessing maybe/yes per [3])

3. the docs refer to IndexUpgraderTool[2] but nothing indicates when/if 
I should be running this manually as far as I can tell. How does this 
come into play?

And of course if there are any docs it sounds like I may have missed (or 
useful threads / issues / whatever) comes to mind that I should read 
feel free to point me at them.

Thanks so much!
-richard

[1]: https://solr.apache.org/guide/8_7/upgrading-a-solr-cluster.html
[2]: https://solr.apache.org/guide/8_7/indexupgrader-tool.html
[3]: 
https://cwiki.apache.org/confluence/display/LUCENE/BackwardsCompatibility#BackwardsCompatibility-FileFormats

Re: Solrcloud v7 -> v8 upgrade question

[Jan Høydahl <ja...@cominvent.com>]

>> The recommendation is however to install a brand new 9.0 cluster
>> (since 8.x will be EOL within a year) and re-index.
> 
> Point taken. Given the seemingly N-1 posture for Solr & Lucene I'm guessing if we wanted to do that we would use the IndexUpgraderTool to convert indexes from 7->8 and then let the v9 deployment handle the 8->9 upgrade?

I think that could work. As far as I remember, it is only Lucene 8 that is going to hard fail if it finds an index originally created by N-2 version, and that Lucene 9 will eat a v8 index, even if upgraded by IndexUpgrader from v7.

> So in this case we've got an index backup from Solr8 that Solr7 is attempting to serve. That seems non ideal / potentially non-functional if Solr has mechanisms that fail when it sees unexpected version info (which it sounds like it does based on your commentary around the IndexUpgraderTool below).

This would definitely be an issue. So best do as follows:

1. Stop all new documents flowing into the cluster
2. Do a last BACKUP call (v7)
3. Complete the rolling upgrade of all nodes to v8
4. Resume indexing traffic
5. Resume normal BACKUP and RESTORE logic

Jan

Re: Solrcloud v7 -> v8 upgrade question

[Richard <so...@okcoder.com>]

Hi Jan!

Really appreciate your follow up. I did have a couple of clarifying 
questions to bounce of you

On 2022-08-14 11:39, Jan Høydahl wrote:
>> 1. The upgrade process[1] suggests an N+/-1 compat story, e.g. 8.11.2 
>> should be able to co-mingle with 7.>3 but I haven't turned up any firm 
>> statements about that.
> 
> Yes, rolling upgrade from 7.x (recommend first going to latest 7.x) to
> 8.x is possible as outlined here
> https://solr.apache.org/guide/solr/latest/upgrade-notes/major-changes-in-solr-8.html#rolling-upgrades-with-solr-8.
> You should carefully audit all the upgrade notes before starting.

Appreciate for the pointer! We had caught the HTTP/1.1 vs HTTP/2 bit 
while reviewing the change list but it's a good reminder :D

> By simulating the full upgrade procedure on your test / QA cluster
> first you can weed out any issues before heading to your prod cluster.

Yup, this bit of reading / research is in service of fleshing out our 
process for the dev fleet. Definitely not going to lead with prod.

> The recommendation is however to install a brand new 9.0 cluster
> (since 8.x will be EOL within a year) and re-index.

Point taken. Given the seemingly N-1 posture for Solr & Lucene I'm 
guessing if we wanted to do that we would use the IndexUpgraderTool to 
convert indexes from 7->8 and then let the v9 deployment handle the 8->9 
upgrade?

>> 2. Is there forward/backward compatibility between segment/index files 
>> between version? e.g. we have automatic core recovery from backups in 
>> the event that a node fails. If I'm running a v8 node that fails after 
>> a backup has been taken and the core gets restored to a v7 does that 
>> sound problematic? (I'm guessing maybe/yes per [3])
> 
> If the restore is an index created with 7.x, no issues, see above.

Right, the scenario I was worried about:

1. NodeA + NodeB are both running Solr 7 (they're just nodes in the 
system, no replication, serve disparate cores)
2. NodeA begins upgrade process, now running Solr 8
3. NodeA takes backup on regular intervals, sends to s3 bucket
4. NodeA host failure
5. Custom cluster automation notices NodeA is dead and identifies it 
should find new node for the cores that were previously assigned to 
NodeA
6. Automation reassigns NodeA cores to NodeB which has not yet been 
upgraded to Solr8
7. NodeB (Solr7) restores index from s3 backups, tries to serve Solr8 
index

So in this case we've got an index backup from Solr8 that Solr7 is 
attempting to serve. That seems non ideal / potentially non-functional 
if Solr has mechanisms that fail when it sees unexpected version info 
(which it sounds like it does based on your commentary around the 
IndexUpgraderTool below).

> [...] then Solr8 would refuse to start since it will find traces
> of v6.x in the binary index files.

-r

Re: Solrcloud v7 -> v8 upgrade question

[Jan Høydahl <ja...@cominvent.com>]

> 1. The upgrade process[1] suggests an N+/-1 compat story, e.g. 8.11.2 should be able to co-mingle with 7.>3 but I haven't turned up any firm statements about that.

Yes, rolling upgrade from 7.x (recommend first going to latest 7.x) to 8.x is possible as outlined here https://solr.apache.org/guide/solr/latest/upgrade-notes/major-changes-in-solr-8.html#rolling-upgrades-with-solr-8. You should carefully audit all the upgrade notes before starting.

However, this assumes your collection was initially made on 7.x. If your index was made on earlier version, e.g. 6.x and then upgraded to 7.x, then you cannot upgrade to 8.x and should re-index from scratch. This is also documented.

By simulating the full upgrade procedure on your test / QA cluster first you can weed out any issues before heading to your prod cluster.

The recommendation is however to install a brand new 9.0 cluster (since 8.x will be EOL within a year) and re-index.

> 2. Is there forward/backward compatibility between segment/index files between version? e.g. we have automatic core recovery from backups in the event that a node fails. If I'm running a v8 node that fails after a backup has been taken and the core gets restored to a v7 does that sound problematic? (I'm guessing maybe/yes per [3])

If the restore is an index created with 7.x, no issues, see above.

> 3. the docs refer to IndexUpgraderTool[2] but nothing indicates when/if I should be running this manually as far as I can tell. How does this come into play?

Don't use that tool. Solr 8 is capable of reading v7 indices, so no need. And if your index was first created in 6.x, and you go ahead using the upgrader tool to get it to 7.x and then 8.x, then Solr8 would refuse to start since it will find traces of v6.x in the binary index files.

> And of course if there are any docs it sounds like I may have missed (or useful threads / issues / whatever) comes to mind that I should read feel free to point me at them.

Most of what you'll need is here
https://solr.apache.org/guide/8_11/solr-upgrade-notes.html
and for 9.0 here
https://solr.apache.org/guide/solr/latest/upgrade-notes/solr-upgrade-notes.html
The reindexing page is also useful
https://solr.apache.org/guide/solr/latest/indexing-guide/reindexing.html
In particular the "Upgrades" section
https://solr.apache.org/guide/solr/latest/indexing-guide/reindexing.html#upgrades

Jan