You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Hrishikesh Gadre <ga...@gmail.com> on 2017/08/11 20:16:52 UTC
Solr config upgrade tool
Hi,
I am currently working on a tool to identify (and in some cases fix) the
incompatibilities in the Solr configuration (e.g. schema.xml,
solrconfig.xml etc.) between major versions of Solr.
My goal is to simplify the Solr upgrade process by providing upgrade
instructions tailored to your configuration. These instuctions can help you
to answer following questions
- Does my Solr configuration have any backwards incompatible sections? If
yes which ones?
- For each of the incompatibility - what do I need to do to fix this
incompatibility? Where can I get more information about why this
incompatibility was introduced (e.g. references to Lucene/Solr jira)?
- Are there any changes in Lucene/Solr which would require me to do a full
reindexing OR can I get away with an index upgrade?
The initial prototype of this tool is available @
https://github.com/hgadre/solr-upgrade-tool
For comments (or suggestions) please refer to
https://issues.apache.org/jira/browse/SOLR-11229
I have implemented upgrade rules for Solr 4 -> Solr 5 and Solr 5 -> Solr 6
by referring to upgrade instructions in the Solr release notes. But during
testing I found that many of the configuration changes (e.g. plugin
deprecations) are not listed in the upgrade section. I am working on
identifying such configuration changes and adding upgrade rules
accordingly. I would love any community participation in improving this
tool.
I will be presenting a talk titled "Apache Solr - Upgrading your upgrade
experience" @ Lucene revolution this year. Please do attend to know more
about simplifying Solr upgrade process.
Regards
Hrishikesh
Re: Solr config upgrade tool
Posted by Hrishikesh Gadre <ga...@gmail.com>.
Interesting. This is certainly useful. I was also thinking about having a
tool which can setup a Solr server of intended version and test out the
upgraded configuration.
The validity of configuration files can be checked by creating a Solr
collection (or core) and verifying that it is loaded successfully.
Regarding indexing and querying - I think we can simplify this by having
user provide a sample data for indexing purpose along with set of queries
(along with the expected results) to try out. The tool then can
automatically index the documents, run queries and verify results.
-Hrishikesh
On Fri, Aug 11, 2017 at 1:29 PM, Davis, Daniel (NIH/NLM) [C] <
daniel.davis@nih.gov> wrote:
> Hrishikesh Gadre,
>
> I'm interested in how that might integrate with continuous integration. I
> briefly worked on a tool to try a configuration out with SolrCloud, e.g.
> upload the config, create a collection, run some stuff, test some stuff. I
> got the first two working, but not the "run some stuff" and "test some
> stuff" parts of this.
>
> It seemed to me that since the main purposes of config are handlers and
> fields, that using SimplePostTool to import data and such would be useful.
> I wanted to stick to Java to really validate that this stuff is really as
> easy to do in Java in Python/Node/Ruby. However, it's not, which means a
> real solution would be opinionated and pick a language.
>
> If curious, you can look at my work at https://github.com/danizen/
> solr-config-tool. The main innovation is to actually use Junit 4 POJO
> files as tests, because TestController creates a configured test case
> containing all of the test cases. The other benefit of making these test
> cases in a Maven project, is that a maven based project that is using the
> solr-config-tool command-line could include these tests explicitly in a
> test plan, and their own tests as well. In this case, they'd parameterize
> their test using properties, and then the SolrConfig object would tell them
> where to find the Solr collection under test.
>
> My intention was for this to help with big-iron investments in Solr, where
> they don't want to spin-up 5 new nodes to test one small schema, which
> would have been the case for my institution.
>
> Maybe I'll give this tool some love soon...
>
> -----Original Message-----
> From: Hrishikesh Gadre [mailto:gadre.solr@gmail.com]
> Sent: Friday, August 11, 2017 4:17 PM
> To: solr-user@lucene.apache.org; dev@lucene.apache.org
> Subject: Solr config upgrade tool
>
> Hi,
>
> I am currently working on a tool to identify (and in some cases fix) the
> incompatibilities in the Solr configuration (e.g. schema.xml,
> solrconfig.xml etc.) between major versions of Solr.
>
> My goal is to simplify the Solr upgrade process by providing upgrade
> instructions tailored to your configuration. These instuctions can help you
> to answer following questions
>
> - Does my Solr configuration have any backwards incompatible sections? If
> yes which ones?
> - For each of the incompatibility - what do I need to do to fix this
> incompatibility? Where can I get more information about why this
> incompatibility was introduced (e.g. references to Lucene/Solr jira)?
> - Are there any changes in Lucene/Solr which would require me to do a full
> reindexing OR can I get away with an index upgrade?
>
> The initial prototype of this tool is available @
> https://github.com/hgadre/solr-upgrade-tool
>
> For comments (or suggestions) please refer to
> https://issues.apache.org/jira/browse/SOLR-11229
>
> I have implemented upgrade rules for Solr 4 -> Solr 5 and Solr 5 -> Solr 6
> by referring to upgrade instructions in the Solr release notes. But during
> testing I found that many of the configuration changes (e.g. plugin
> deprecations) are not listed in the upgrade section. I am working on
> identifying such configuration changes and adding upgrade rules
> accordingly. I would love any community participation in improving this
> tool.
>
> I will be presenting a talk titled "Apache Solr - Upgrading your upgrade
> experience" @ Lucene revolution this year. Please do attend to know more
> about simplifying Solr upgrade process.
>
> Regards
>
> Hrishikesh
>
RE: Solr config upgrade tool
Posted by "Davis, Daniel (NIH/NLM) [C]" <da...@nih.gov>.
Hrishikesh Gadre,
I'm interested in how that might integrate with continuous integration. I briefly worked on a tool to try a configuration out with SolrCloud, e.g. upload the config, create a collection, run some stuff, test some stuff. I got the first two working, but not the "run some stuff" and "test some stuff" parts of this.
It seemed to me that since the main purposes of config are handlers and fields, that using SimplePostTool to import data and such would be useful. I wanted to stick to Java to really validate that this stuff is really as easy to do in Java in Python/Node/Ruby. However, it's not, which means a real solution would be opinionated and pick a language.
If curious, you can look at my work at https://github.com/danizen/solr-config-tool. The main innovation is to actually use Junit 4 POJO files as tests, because TestController creates a configured test case containing all of the test cases. The other benefit of making these test cases in a Maven project, is that a maven based project that is using the solr-config-tool command-line could include these tests explicitly in a test plan, and their own tests as well. In this case, they'd parameterize their test using properties, and then the SolrConfig object would tell them where to find the Solr collection under test.
My intention was for this to help with big-iron investments in Solr, where they don't want to spin-up 5 new nodes to test one small schema, which would have been the case for my institution.
Maybe I'll give this tool some love soon...
-----Original Message-----
From: Hrishikesh Gadre [mailto:gadre.solr@gmail.com]
Sent: Friday, August 11, 2017 4:17 PM
To: solr-user@lucene.apache.org; dev@lucene.apache.org
Subject: Solr config upgrade tool
Hi,
I am currently working on a tool to identify (and in some cases fix) the incompatibilities in the Solr configuration (e.g. schema.xml, solrconfig.xml etc.) between major versions of Solr.
My goal is to simplify the Solr upgrade process by providing upgrade instructions tailored to your configuration. These instuctions can help you to answer following questions
- Does my Solr configuration have any backwards incompatible sections? If yes which ones?
- For each of the incompatibility - what do I need to do to fix this incompatibility? Where can I get more information about why this incompatibility was introduced (e.g. references to Lucene/Solr jira)?
- Are there any changes in Lucene/Solr which would require me to do a full reindexing OR can I get away with an index upgrade?
The initial prototype of this tool is available @ https://github.com/hgadre/solr-upgrade-tool
For comments (or suggestions) please refer to
https://issues.apache.org/jira/browse/SOLR-11229
I have implemented upgrade rules for Solr 4 -> Solr 5 and Solr 5 -> Solr 6 by referring to upgrade instructions in the Solr release notes. But during testing I found that many of the configuration changes (e.g. plugin
deprecations) are not listed in the upgrade section. I am working on identifying such configuration changes and adding upgrade rules accordingly. I would love any community participation in improving this tool.
I will be presenting a talk titled "Apache Solr - Upgrading your upgrade experience" @ Lucene revolution this year. Please do attend to know more about simplifying Solr upgrade process.
Regards
Hrishikesh