You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Shawn Heisey <ap...@elyograg.org> on 2016/09/25 17:41:55 UTC
Solr configuration format fracturing
There seems to be some fracturing in the format of various Solr
configs. Most of the config uses XML, but some new features in the last
few years are using JSON, particularly where SolrCloud and Zookeeper are
concerned. When notifications about SOLR-9557 came through, it revealed
that there is a config file sitting next to solrconfig.xml named
"params.json" that Solr will use. I wasn't aware of this until reading
that issue.
This leads me to suggest something rather drastic for 7.0: Consolidate
all configuration formats and agree to consistent format usage unless
there is another major discussion and agreement to change formats.
I did consider starting this discussion in Jira, but it's fairly major,
so the dev list seemed like the right place to start.
Comments from some new users have come my way along the lines of "XML is
so 90's ... get with the times!" Image problems like that can be fatal
to a software project, even if there's no technical problem.
The likely winner in the format discussion is pure unmodified JSON, but
I'm not going to make any assumptions. SOLR-8029 has some format
discussions that may be relevant here.
IMHO, in order to make the idea successful, Solr 7.0 will need to
automatically convert most configs on startup from the old format to the
new format without user intervention. If there's something that we find
we can't convert automatically, that should result in a failure to
start, with a helpful message so the user has some idea what they need
to do.
Thoughts? Is this too scary to contemplate? Should I open an umbrella
issue in Jira to get the ball rolling?
Thanks,
Shawn
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr configuration format fracturing
Posted by Noble Paul <no...@gmail.com>.
I agree in principle that there should be a way to download,store and
restore configuration.
Isn't it better achieved through the configset API?
like
bin/solr save-config gettingstarted <gettingstarted.zip>
bin/solr restore-config gettingstarted.zip gettingstarted
The problem with saving a single file is that most of them are
interdependent and saving a single file alone will break stuff.
Users still can download and save a single file from the
/admin/zookeeper path. I know it is not very friendly today but we can
make it better.
On Mon, Sep 26, 2016 at 1:19 PM, Jan Høydahl <ja...@cominvent.com> wrote:
> I agree that configuring Solr through the APIs is a better option than
> uploading huge XMLs, but we should strive for copy/paste-ability wherever
> possible. Ideally it should be possible to store your config/schema
> information
> in JSON format in GIT and easily POST it after creating a new collection.
> The exact same JSON should also be possible to paste into the Admin UI
> in order to reproduce the same config/schema. And if you do a GET request
> to fetch config/schema, the resulting JSON should ideally be in a format
> that
> can be again checked into GIT and work when re-creating the same collection.
>
> Am I right that the bulk API style of embedding the verb into the data
> object
> itself, e.g. "add-field-type”, is the main source of the difference here, or
> is it more
> complex than that? Looks to me that the JSON object after “add-field-type”
> and
> “replace-field-type” is exactly the same as the one you get from a call to
> GET /solr/collection/schema/fieldtypes/name?omitHeader=true except from
> the latter being wrapped in a “fieldType” object.
>
> I’m not totally against the “add-field-type” bulk style API, but perhaps we
> should also have a GET/POST /solr/collection/schema endpoint which
> supports roundtrip of the whole schema in JSON format?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> 26. sep. 2016 kl. 09.10 skrev Noble Paul <no...@gmail.com>:
>
> As the main culprit here let me put forward my few cents
>
> * The intent is to go away from config files. Users should use APIs to
> read and edit configurations. Editing configuration is both error
> prone and old school
> * I prefer to store everything is JSON. json is modern and simple.
> * Even today it is totally possible to put an empty solrconfig.xml and
> move everything to configoverlay.json. We may not need to wait for a
> big release to do that
> * schema is still in xml . there is significant effort involved in
> changing it over
> * There is a reason why the /config output has solrconfig.xml and
> overlay.json merged because they don't make sense individually. params
> are not relevant unless there is a reference to it in the config. If a
> requesthandler refers to a paramset, we probably should show it inline
> * Yes , you can't round-trip the output of config. I thought of adding
> a wt which emits a "round-trippable" JSON (like the SHOW CREATE TABLE
> functionality in RDBMS)
>
> On Mon, Sep 26, 2016 at 5:35 AM, Alexandre Rafalovitch
> <ar...@gmail.com> wrote:
>
> Did you know about configoverlay.json?
>
> +1 to the discussion.
>
> Additional fuel to the fire is that /config endpoint will return
> solrconfig.xml + overlay.json merged, but not params.json. Confusing.
>
> Additionally, /config output is JSON but not one that can round-trip AFAIK.
>
> Regards,
> Alex
>
>
> On 26 Sep 2016 12:42 AM, "Shawn Heisey" <ap...@elyograg.org> wrote:
>
>
> There seems to be some fracturing in the format of various Solr
> configs. Most of the config uses XML, but some new features in the last
> few years are using JSON, particularly where SolrCloud and Zookeeper are
> concerned. When notifications about SOLR-9557 came through, it revealed
> that there is a config file sitting next to solrconfig.xml named
> "params.json" that Solr will use. I wasn't aware of this until reading
> that issue.
>
> This leads me to suggest something rather drastic for 7.0: Consolidate
> all configuration formats and agree to consistent format usage unless
> there is another major discussion and agreement to change formats.
>
> I did consider starting this discussion in Jira, but it's fairly major,
> so the dev list seemed like the right place to start.
>
> Comments from some new users have come my way along the lines of "XML is
> so 90's ... get with the times!" Image problems like that can be fatal
> to a software project, even if there's no technical problem.
>
> The likely winner in the format discussion is pure unmodified JSON, but
> I'm not going to make any assumptions. SOLR-8029 has some format
> discussions that may be relevant here.
>
> IMHO, in order to make the idea successful, Solr 7.0 will need to
> automatically convert most configs on startup from the old format to the
> new format without user intervention. If there's something that we find
> we can't convert automatically, that should result in a failure to
> start, with a helpful message so the user has some idea what they need
> to do.
>
> Thoughts? Is this too scary to contemplate? Should I open an umbrella
> issue in Jira to get the ball rolling?
>
> Thanks,
> Shawn
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
--
-----------------------------------------------------
Noble Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr configuration format fracturing
Posted by Jan Høydahl <ja...@cominvent.com>.
I agree that configuring Solr through the APIs is a better option than
uploading huge XMLs, but we should strive for copy/paste-ability wherever
possible. Ideally it should be possible to store your config/schema information
in JSON format in GIT and easily POST it after creating a new collection.
The exact same JSON should also be possible to paste into the Admin UI
in order to reproduce the same config/schema. And if you do a GET request
to fetch config/schema, the resulting JSON should ideally be in a format that
can be again checked into GIT and work when re-creating the same collection.
Am I right that the bulk API style of embedding the verb into the data object
itself, e.g. "add-field-type”, is the main source of the difference here, or is it more
complex than that? Looks to me that the JSON object after “add-field-type” and
“replace-field-type” is exactly the same as the one you get from a call to
GET /solr/collection/schema/fieldtypes/name?omitHeader=true except from
the latter being wrapped in a “fieldType” object.
I’m not totally against the “add-field-type” bulk style API, but perhaps we
should also have a GET/POST /solr/collection/schema endpoint which
supports roundtrip of the whole schema in JSON format?
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
> 26. sep. 2016 kl. 09.10 skrev Noble Paul <no...@gmail.com>:
>
> As the main culprit here let me put forward my few cents
>
> * The intent is to go away from config files. Users should use APIs to
> read and edit configurations. Editing configuration is both error
> prone and old school
> * I prefer to store everything is JSON. json is modern and simple.
> * Even today it is totally possible to put an empty solrconfig.xml and
> move everything to configoverlay.json. We may not need to wait for a
> big release to do that
> * schema is still in xml . there is significant effort involved in
> changing it over
> * There is a reason why the /config output has solrconfig.xml and
> overlay.json merged because they don't make sense individually. params
> are not relevant unless there is a reference to it in the config. If a
> requesthandler refers to a paramset, we probably should show it inline
> * Yes , you can't round-trip the output of config. I thought of adding
> a wt which emits a "round-trippable" JSON (like the SHOW CREATE TABLE
> functionality in RDBMS)
>
> On Mon, Sep 26, 2016 at 5:35 AM, Alexandre Rafalovitch
> <ar...@gmail.com> wrote:
>> Did you know about configoverlay.json?
>>
>> +1 to the discussion.
>>
>> Additional fuel to the fire is that /config endpoint will return
>> solrconfig.xml + overlay.json merged, but not params.json. Confusing.
>>
>> Additionally, /config output is JSON but not one that can round-trip AFAIK.
>>
>> Regards,
>> Alex
>>
>>
>> On 26 Sep 2016 12:42 AM, "Shawn Heisey" <ap...@elyograg.org> wrote:
>>>
>>> There seems to be some fracturing in the format of various Solr
>>> configs. Most of the config uses XML, but some new features in the last
>>> few years are using JSON, particularly where SolrCloud and Zookeeper are
>>> concerned. When notifications about SOLR-9557 came through, it revealed
>>> that there is a config file sitting next to solrconfig.xml named
>>> "params.json" that Solr will use. I wasn't aware of this until reading
>>> that issue.
>>>
>>> This leads me to suggest something rather drastic for 7.0: Consolidate
>>> all configuration formats and agree to consistent format usage unless
>>> there is another major discussion and agreement to change formats.
>>>
>>> I did consider starting this discussion in Jira, but it's fairly major,
>>> so the dev list seemed like the right place to start.
>>>
>>> Comments from some new users have come my way along the lines of "XML is
>>> so 90's ... get with the times!" Image problems like that can be fatal
>>> to a software project, even if there's no technical problem.
>>>
>>> The likely winner in the format discussion is pure unmodified JSON, but
>>> I'm not going to make any assumptions. SOLR-8029 has some format
>>> discussions that may be relevant here.
>>>
>>> IMHO, in order to make the idea successful, Solr 7.0 will need to
>>> automatically convert most configs on startup from the old format to the
>>> new format without user intervention. If there's something that we find
>>> we can't convert automatically, that should result in a failure to
>>> start, with a helpful message so the user has some idea what they need
>>> to do.
>>>
>>> Thoughts? Is this too scary to contemplate? Should I open an umbrella
>>> issue in Jira to get the ball rolling?
>>>
>>> Thanks,
>>> Shawn
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
Re: Solr configuration format fracturing
Posted by Noble Paul <no...@gmail.com>.
We had a very large monolithic solrconfig.xml which users used to
(they still do) hand edit. This ws fine in standalone mode and in
solrcloud it's unwieldy. Imagine you making a single typo and blowing
up an entire sollection with hundreds of nodes. When we change a
single parameter, it resulted in reload of all these cores which would
lead to performance degradation across the cluster. The other problem
is that very little of solrconfig.xml was really 'user code' it was
mostly system code. It was 95% dictated by solr (and never changed).
So these solutions were important as we moved from the simple
standlone server to Solr cloud.
Now, the question is how do we mitigate this problem and educate users
with the new 'way of life'
We must remove more and more stuff from solrconfig.xml. The strategy
is as follows. Every component should be available implicitly and it
should be accessible through request parameter flags. The rest should
move to the configoverlay.json.
The bigger issue is , how do we make users comfortable with it. It
requires a lot more documentation and many more user-friendly APIs
On Mon, Sep 26, 2016 at 1:26 PM, Alexandre Rafalovitch
<ar...@gmail.com> wrote:
> I think each individual change mostly made sense. What I think becomes
> a problem is expecting the user to understand the end result when they
> are not aware of what different things have and where they live.
>
> solrconfig.xml is a very large beast already. But, until recently,
> there was a sense that one could read and/or grep for whatever one did
> not understand (request handler name, parameter name, etc) in one of
> those two files (solrconfig.xml, schema.xml) and have the answer.
>
> Now, we have 3 files and two API calls for solrconfig.xml, two names
> for schema (yes, schema.xml given all _other_ tutorials on the web)
> and whatever the magic _rest_managed.json file is for meta-managing
> managed components. Plus, the implicit handlers, and now implicit
> initParam names for implicit handlers :-) And, still dark magic to me,
> the order of invocation between parameter information passed through
> the request handler, initParams, params.json, request params directly,
> request params mentioning params groups, etc (ok, I am rambling here).
>
> I think magic is not bad (that's a recent change of mind frankly), if
> that significantly simplifies workflow and still allows for easy
> debugging. But I am not sure the current state - taken all together -
> achieves either of two purposes.
>
> I think looking at all these different thinks together, understanding
> the use cases and perhaps slimming down to a unified forward-back path
> would be best. So, a single format (JSON if it works,
> JSON+comments/quotes that Solr actually supports, or whatever). But
> then the inputs and outputs should be consistent so that the same
> tooling could be used for everything.
>
> Too much to dream about?
>
> Regards,
> Alex.
> P.s. Oh, and minimum solrconfig.xml requires luceneMatchVersion to be
> present. And possibly a /select handler. But yes, a valid core
> configuration can be very very small.
>
> ----
> Newsletter and resources for Solr beginners and intermediates:
> http://www.solr-start.com/
>
>
> On 26 September 2016 at 14:10, Noble Paul <no...@gmail.com> wrote:
>> As the main culprit here let me put forward my few cents
>>
>> * The intent is to go away from config files. Users should use APIs to
>> read and edit configurations. Editing configuration is both error
>> prone and old school
>> * I prefer to store everything is JSON. json is modern and simple.
>> * Even today it is totally possible to put an empty solrconfig.xml and
>> move everything to configoverlay.json. We may not need to wait for a
>> big release to do that
>> * schema is still in xml . there is significant effort involved in
>> changing it over
>> * There is a reason why the /config output has solrconfig.xml and
>> overlay.json merged because they don't make sense individually. params
>> are not relevant unless there is a reference to it in the config. If a
>> requesthandler refers to a paramset, we probably should show it inline
>> * Yes , you can't round-trip the output of config. I thought of adding
>> a wt which emits a "round-trippable" JSON (like the SHOW CREATE TABLE
>> functionality in RDBMS)
>>
>> On Mon, Sep 26, 2016 at 5:35 AM, Alexandre Rafalovitch
>> <ar...@gmail.com> wrote:
>>> Did you know about configoverlay.json?
>>>
>>> +1 to the discussion.
>>>
>>> Additional fuel to the fire is that /config endpoint will return
>>> solrconfig.xml + overlay.json merged, but not params.json. Confusing.
>>>
>>> Additionally, /config output is JSON but not one that can round-trip AFAIK.
>>>
>>> Regards,
>>> Alex
>>>
>>>
>>> On 26 Sep 2016 12:42 AM, "Shawn Heisey" <ap...@elyograg.org> wrote:
>>>>
>>>> There seems to be some fracturing in the format of various Solr
>>>> configs. Most of the config uses XML, but some new features in the last
>>>> few years are using JSON, particularly where SolrCloud and Zookeeper are
>>>> concerned. When notifications about SOLR-9557 came through, it revealed
>>>> that there is a config file sitting next to solrconfig.xml named
>>>> "params.json" that Solr will use. I wasn't aware of this until reading
>>>> that issue.
>>>>
>>>> This leads me to suggest something rather drastic for 7.0: Consolidate
>>>> all configuration formats and agree to consistent format usage unless
>>>> there is another major discussion and agreement to change formats.
>>>>
>>>> I did consider starting this discussion in Jira, but it's fairly major,
>>>> so the dev list seemed like the right place to start.
>>>>
>>>> Comments from some new users have come my way along the lines of "XML is
>>>> so 90's ... get with the times!" Image problems like that can be fatal
>>>> to a software project, even if there's no technical problem.
>>>>
>>>> The likely winner in the format discussion is pure unmodified JSON, but
>>>> I'm not going to make any assumptions. SOLR-8029 has some format
>>>> discussions that may be relevant here.
>>>>
>>>> IMHO, in order to make the idea successful, Solr 7.0 will need to
>>>> automatically convert most configs on startup from the old format to the
>>>> new format without user intervention. If there's something that we find
>>>> we can't convert automatically, that should result in a failure to
>>>> start, with a helpful message so the user has some idea what they need
>>>> to do.
>>>>
>>>> Thoughts? Is this too scary to contemplate? Should I open an umbrella
>>>> issue in Jira to get the ball rolling?
>>>>
>>>> Thanks,
>>>> Shawn
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
>>>
>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
--
-----------------------------------------------------
Noble Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr configuration format fracturing
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Mon, 2016-09-26 at 14:56 +0700, Alexandre Rafalovitch wrote:
> So, a single format (JSON if it works,
> JSON+comments/quotes that Solr actually supports, or whatever). But
> then the inputs and outputs should be consistent so that the same
> tooling could be used for everything.
This is a very important part. Whatever is chosen should support inline
documentation. In a way, choosing the comment-lacking JSON format is a
good thing as that allow the introduction of an explicit documentation
key:
{ subheadline: {
"_documentation", "Only used for articles from source X, remove if
that source is deprecated",
"type", "ImportantText",
"stored", "true",
...
},
...
}
It would be great to have the content from such a documentation field
pop up in the schema browser in the GUI.
- Toke Eskildsen, State and University Library, Denmark
Re: Solr configuration format fracturing
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
I think each individual change mostly made sense. What I think becomes
a problem is expecting the user to understand the end result when they
are not aware of what different things have and where they live.
solrconfig.xml is a very large beast already. But, until recently,
there was a sense that one could read and/or grep for whatever one did
not understand (request handler name, parameter name, etc) in one of
those two files (solrconfig.xml, schema.xml) and have the answer.
Now, we have 3 files and two API calls for solrconfig.xml, two names
for schema (yes, schema.xml given all _other_ tutorials on the web)
and whatever the magic _rest_managed.json file is for meta-managing
managed components. Plus, the implicit handlers, and now implicit
initParam names for implicit handlers :-) And, still dark magic to me,
the order of invocation between parameter information passed through
the request handler, initParams, params.json, request params directly,
request params mentioning params groups, etc (ok, I am rambling here).
I think magic is not bad (that's a recent change of mind frankly), if
that significantly simplifies workflow and still allows for easy
debugging. But I am not sure the current state - taken all together -
achieves either of two purposes.
I think looking at all these different thinks together, understanding
the use cases and perhaps slimming down to a unified forward-back path
would be best. So, a single format (JSON if it works,
JSON+comments/quotes that Solr actually supports, or whatever). But
then the inputs and outputs should be consistent so that the same
tooling could be used for everything.
Too much to dream about?
Regards,
Alex.
P.s. Oh, and minimum solrconfig.xml requires luceneMatchVersion to be
present. And possibly a /select handler. But yes, a valid core
configuration can be very very small.
----
Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/
On 26 September 2016 at 14:10, Noble Paul <no...@gmail.com> wrote:
> As the main culprit here let me put forward my few cents
>
> * The intent is to go away from config files. Users should use APIs to
> read and edit configurations. Editing configuration is both error
> prone and old school
> * I prefer to store everything is JSON. json is modern and simple.
> * Even today it is totally possible to put an empty solrconfig.xml and
> move everything to configoverlay.json. We may not need to wait for a
> big release to do that
> * schema is still in xml . there is significant effort involved in
> changing it over
> * There is a reason why the /config output has solrconfig.xml and
> overlay.json merged because they don't make sense individually. params
> are not relevant unless there is a reference to it in the config. If a
> requesthandler refers to a paramset, we probably should show it inline
> * Yes , you can't round-trip the output of config. I thought of adding
> a wt which emits a "round-trippable" JSON (like the SHOW CREATE TABLE
> functionality in RDBMS)
>
> On Mon, Sep 26, 2016 at 5:35 AM, Alexandre Rafalovitch
> <ar...@gmail.com> wrote:
>> Did you know about configoverlay.json?
>>
>> +1 to the discussion.
>>
>> Additional fuel to the fire is that /config endpoint will return
>> solrconfig.xml + overlay.json merged, but not params.json. Confusing.
>>
>> Additionally, /config output is JSON but not one that can round-trip AFAIK.
>>
>> Regards,
>> Alex
>>
>>
>> On 26 Sep 2016 12:42 AM, "Shawn Heisey" <ap...@elyograg.org> wrote:
>>>
>>> There seems to be some fracturing in the format of various Solr
>>> configs. Most of the config uses XML, but some new features in the last
>>> few years are using JSON, particularly where SolrCloud and Zookeeper are
>>> concerned. When notifications about SOLR-9557 came through, it revealed
>>> that there is a config file sitting next to solrconfig.xml named
>>> "params.json" that Solr will use. I wasn't aware of this until reading
>>> that issue.
>>>
>>> This leads me to suggest something rather drastic for 7.0: Consolidate
>>> all configuration formats and agree to consistent format usage unless
>>> there is another major discussion and agreement to change formats.
>>>
>>> I did consider starting this discussion in Jira, but it's fairly major,
>>> so the dev list seemed like the right place to start.
>>>
>>> Comments from some new users have come my way along the lines of "XML is
>>> so 90's ... get with the times!" Image problems like that can be fatal
>>> to a software project, even if there's no technical problem.
>>>
>>> The likely winner in the format discussion is pure unmodified JSON, but
>>> I'm not going to make any assumptions. SOLR-8029 has some format
>>> discussions that may be relevant here.
>>>
>>> IMHO, in order to make the idea successful, Solr 7.0 will need to
>>> automatically convert most configs on startup from the old format to the
>>> new format without user intervention. If there's something that we find
>>> we can't convert automatically, that should result in a failure to
>>> start, with a helpful message so the user has some idea what they need
>>> to do.
>>>
>>> Thoughts? Is this too scary to contemplate? Should I open an umbrella
>>> issue in Jira to get the ball rolling?
>>>
>>> Thanks,
>>> Shawn
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr configuration format fracturing
Posted by Noble Paul <no...@gmail.com>.
As the main culprit here let me put forward my few cents
* The intent is to go away from config files. Users should use APIs to
read and edit configurations. Editing configuration is both error
prone and old school
* I prefer to store everything is JSON. json is modern and simple.
* Even today it is totally possible to put an empty solrconfig.xml and
move everything to configoverlay.json. We may not need to wait for a
big release to do that
* schema is still in xml . there is significant effort involved in
changing it over
* There is a reason why the /config output has solrconfig.xml and
overlay.json merged because they don't make sense individually. params
are not relevant unless there is a reference to it in the config. If a
requesthandler refers to a paramset, we probably should show it inline
* Yes , you can't round-trip the output of config. I thought of adding
a wt which emits a "round-trippable" JSON (like the SHOW CREATE TABLE
functionality in RDBMS)
On Mon, Sep 26, 2016 at 5:35 AM, Alexandre Rafalovitch
<ar...@gmail.com> wrote:
> Did you know about configoverlay.json?
>
> +1 to the discussion.
>
> Additional fuel to the fire is that /config endpoint will return
> solrconfig.xml + overlay.json merged, but not params.json. Confusing.
>
> Additionally, /config output is JSON but not one that can round-trip AFAIK.
>
> Regards,
> Alex
>
>
> On 26 Sep 2016 12:42 AM, "Shawn Heisey" <ap...@elyograg.org> wrote:
>>
>> There seems to be some fracturing in the format of various Solr
>> configs. Most of the config uses XML, but some new features in the last
>> few years are using JSON, particularly where SolrCloud and Zookeeper are
>> concerned. When notifications about SOLR-9557 came through, it revealed
>> that there is a config file sitting next to solrconfig.xml named
>> "params.json" that Solr will use. I wasn't aware of this until reading
>> that issue.
>>
>> This leads me to suggest something rather drastic for 7.0: Consolidate
>> all configuration formats and agree to consistent format usage unless
>> there is another major discussion and agreement to change formats.
>>
>> I did consider starting this discussion in Jira, but it's fairly major,
>> so the dev list seemed like the right place to start.
>>
>> Comments from some new users have come my way along the lines of "XML is
>> so 90's ... get with the times!" Image problems like that can be fatal
>> to a software project, even if there's no technical problem.
>>
>> The likely winner in the format discussion is pure unmodified JSON, but
>> I'm not going to make any assumptions. SOLR-8029 has some format
>> discussions that may be relevant here.
>>
>> IMHO, in order to make the idea successful, Solr 7.0 will need to
>> automatically convert most configs on startup from the old format to the
>> new format without user intervention. If there's something that we find
>> we can't convert automatically, that should result in a failure to
>> start, with a helpful message so the user has some idea what they need
>> to do.
>>
>> Thoughts? Is this too scary to contemplate? Should I open an umbrella
>> issue in Jira to get the ball rolling?
>>
>> Thanks,
>> Shawn
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
--
-----------------------------------------------------
Noble Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr configuration format fracturing
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Did you know about configoverlay.json?
+1 to the discussion.
Additional fuel to the fire is that /config endpoint will return
solrconfig.xml + overlay.json merged, but not params.json. Confusing.
Additionally, /config output is JSON but not one that can round-trip AFAIK.
Regards,
Alex
On 26 Sep 2016 12:42 AM, "Shawn Heisey" <ap...@elyograg.org> wrote:
> There seems to be some fracturing in the format of various Solr
> configs. Most of the config uses XML, but some new features in the last
> few years are using JSON, particularly where SolrCloud and Zookeeper are
> concerned. When notifications about SOLR-9557 came through, it revealed
> that there is a config file sitting next to solrconfig.xml named
> "params.json" that Solr will use. I wasn't aware of this until reading
> that issue.
>
> This leads me to suggest something rather drastic for 7.0: Consolidate
> all configuration formats and agree to consistent format usage unless
> there is another major discussion and agreement to change formats.
>
> I did consider starting this discussion in Jira, but it's fairly major,
> so the dev list seemed like the right place to start.
>
> Comments from some new users have come my way along the lines of "XML is
> so 90's ... get with the times!" Image problems like that can be fatal
> to a software project, even if there's no technical problem.
>
> The likely winner in the format discussion is pure unmodified JSON, but
> I'm not going to make any assumptions. SOLR-8029 has some format
> discussions that may be relevant here.
>
> IMHO, in order to make the idea successful, Solr 7.0 will need to
> automatically convert most configs on startup from the old format to the
> new format without user intervention. If there's something that we find
> we can't convert automatically, that should result in a failure to
> start, with a helpful message so the user has some idea what they need
> to do.
>
> Thoughts? Is this too scary to contemplate? Should I open an umbrella
> issue in Jira to get the ball rolling?
>
> Thanks,
> Shawn
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Solr configuration format fracturing
Posted by Noble Paul <no...@gmail.com>.
It's already there SOLR-9569
On Wed, Sep 28, 2016 at 11:59 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 9/25/2016 11:41 AM, Shawn Heisey wrote:
> > Thoughts? Is this too scary to contemplate? Should I open an umbrella
> > issue in Jira to get the ball rolling?
>
> I think we've had a decent discussion, with an indication that we need
> to at least consider some action.
>
> JSON still looks like the winner.
>
> Unless people think it's a bad idea, I plan to file an umbrella issue in
> SOLR for planning and discussion later this evening.
>
> Thanks,
> Shawn
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
--
-----------------------------------------------------
Noble Paul
Re: Solr configuration format fracturing
Posted by Shawn Heisey <ap...@elyograg.org>.
On 9/25/2016 11:41 AM, Shawn Heisey wrote:
> Thoughts? Is this too scary to contemplate? Should I open an umbrella
> issue in Jira to get the ball rolling?
I think we've had a decent discussion, with an indication that we need
to at least consider some action.
JSON still looks like the winner.
Unless people think it's a bad idea, I plan to file an umbrella issue in
SOLR for planning and discussion later this evening.
Thanks,
Shawn
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org