[jira] [Created] (LUCENE-9234) Keep write support for old codecs?

["Adrien Grand (Jira)" <ji...@apache.org>]

Adrien Grand created LUCENE-9234:
------------------------------------

             Summary: Keep write support for old codecs?
                 Key: LUCENE-9234
                 URL: https://issues.apache.org/jira/browse/LUCENE-9234
             Project: Lucene - Core
          Issue Type: Wish
            Reporter: Adrien Grand


Currenty we maintain read/write support for the latest codec in lucene/core, and read-only support for codecs of previous versions (up to \{N-1\}.0}) in  lucene/backward-codecs. We often keep write support in test-framework for testing purposes only.

This raises challenges for Elasticsearch with regard to rolling upgrades: we have some users who index very large amounts of data on clusters that are quite large, so that rolling upgrades take significant time. Meanwhile, several indices may be created.

Allocating indices when the cluster has nodes of different versions requires care as Lucene indices created on nodes with a newer version cannot be read by the nodes running the older version. It is possible to force primary replicas to be allocated on the older nodes, but this brings other problems like availability, uneven disk usage across nodes, or moving a lot of data around.

If Lucene could write data using the minimum version that exists in the cluster, this would avoid this problem as the written data could be read by any node of the cluster. I understand this change would not come for free, especially when it comes to testing as we'd need to make sure that older Lucene versions can read indices created by this "compatibility mode".

I'd be curious to understand whether this is a problem for Solr too, if not how this problem is being handled, and maybe whether there are other problems that you have encountered that would also benefit from the ability to write data with an older format.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org