You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by David Smiley <ds...@apache.org> on 2021/03/20 04:36:02 UTC
Deprecation of QueryElevationComponent's support for elevate.xml in
data directories
Hey Solr community,
I found myself doing some maintenance inside QueryElevationComponent
related to some much larger refactoring. I'm seeing a fair amount of
complexity around it's support of "elevate.xml" in a Solr *data* directory
(where the index is). I don't know of any other Solr component that does
anything like this. This feature of QEC existed from the start in Jan
2008. In October of that same year, Solr's replication handler gained the
ability to replicate config files (see "confFiles" option) between cores.
I think this aspect of QEC would not have existed if it came later. In
SolrCloud, you're expected to use ZooKeeper of course.
I plan to remove this mechanism in 9.0 on the grounds that it is obsolete.
To account for the ability to automatically detect elevate.xml changes in a
data dir and reload on a commit, I'm adding this mechanism to the
configSet/conf. I'm not doing so for SolrCloud at this time because of
overhead concerns but it could be added later with some work.
I also plan to remove QEC's support of the elevate.xml file to be
"versioned" -- e.g. elevate.xml.001, elevate.xml.002. The underlying
mechanism was added in
https://issues.apache.org/jira/browse/SOLR-351 where it was argued to be
useful for Windows OS which cannot replace an already open file. Obviously
this is a non-issue in SolrCloud, and perhaps not an issue if you use the
confFiles replication feature to edit it either. Regardless, I'm highly
suspicious it's an issue whatsoever since this file shouldn't be held open
on a sustained basis; it's only loaded on core load / replication and with
my changes above, on a commit if the modification time changes.
If you have concerns, let me know.
SOLR-15274 - QueryElevationComponent: auto-load file changes; remove data
dir support <https://issues.apache.org/jira/browse/SOLR-15274>
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
Fwd: Deprecation of QueryElevationComponent's support for elevate.xml
in data directories
Posted by David Smiley <ds...@apache.org>.
---------- Forwarded message ---------
From: David Smiley <ds...@apache.org>
Date: Sat, Mar 20, 2021 at 12:36 AM
Subject: Deprecation of QueryElevationComponent's support for elevate.xml
in data directories
To: solr-user <so...@lucene.apache.org>
Cc: Bruno Roustant <br...@gmail.com>, Nazerke Seidan <
seinaz1997@gmail.com>, <mo...@europeana.eu>
Hey Solr community,
I found myself doing some maintenance inside QueryElevationComponent
related to some much larger refactoring. I'm seeing a fair amount of
complexity around it's support of "elevate.xml" in a Solr *data* directory
(where the index is). I don't know of any other Solr component that does
anything like this. This feature of QEC existed from the start in Jan
2008. In October of that same year, Solr's replication handler gained the
ability to replicate config files (see "confFiles" option) between cores.
I think this aspect of QEC would not have existed if it came later. In
SolrCloud, you're expected to use ZooKeeper of course.
I plan to remove this mechanism in 9.0 on the grounds that it is obsolete.
To account for the ability to automatically detect elevate.xml changes in a
data dir and reload on a commit, I'm adding this mechanism to the
configSet/conf. I'm not doing so for SolrCloud at this time because of
overhead concerns but it could be added later with some work.
I also plan to remove QEC's support of the elevate.xml file to be
"versioned" -- e.g. elevate.xml.001, elevate.xml.002. The underlying
mechanism was added in
https://issues.apache.org/jira/browse/SOLR-351 where it was argued to be
useful for Windows OS which cannot replace an already open file. Obviously
this is a non-issue in SolrCloud, and perhaps not an issue if you use the
confFiles replication feature to edit it either. Regardless, I'm highly
suspicious it's an issue whatsoever since this file shouldn't be held open
on a sustained basis; it's only loaded on core load / replication and with
my changes above, on a commit if the modification time changes.
If you have concerns, let me know.
SOLR-15274 - QueryElevationComponent: auto-load file changes; remove data
dir support <https://issues.apache.org/jira/browse/SOLR-15274>
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
Re: Deprecation of QueryElevationComponent's support for elevate.xml
in data directories
Posted by David Smiley <ds...@apache.org>.
A commit is required or else there might be dirty cache problems if there
were cached queries affected by QEC.. Generally, Solr should always return
the same result if there hasn't been a commit. I know there are
short-circuit optimizations for commits to no-op if there's no new data so
I could see a need for either forcing it or for Solr to somehow "watch" the
configs to do a commit on that basis.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Mon, Mar 22, 2021 at 4:28 AM Mónica Marrero <mo...@europeana.eu>
wrote:
> It seems to be a cleaner way to manage this. Just a comment: what about a
> mechanism to just update this file in memory? I know it will also involve
> opening a new searcher, but at least It does not depend on commits in the
> data (we may need to force changes in the data when what we really want to
> do is to update the elevate.xml file).
>
>
> On Sat, 20 Mar 2021 at 05:36, David Smiley <ds...@apache.org> wrote:
>
>> Hey Solr community,
>>
>> I found myself doing some maintenance inside QueryElevationComponent
>> related to some much larger refactoring. I'm seeing a fair amount of
>> complexity around it's support of "elevate.xml" in a Solr *data* directory
>> (where the index is). I don't know of any other Solr component that does
>> anything like this. This feature of QEC existed from the start in Jan
>> 2008. In October of that same year, Solr's replication handler gained the
>> ability to replicate config files (see "confFiles" option) between cores.
>> I think this aspect of QEC would not have existed if it came later. In
>> SolrCloud, you're expected to use ZooKeeper of course.
>>
>> I plan to remove this mechanism in 9.0 on the grounds that it is obsolete.
>>
>> To account for the ability to automatically detect elevate.xml changes in
>> a data dir and reload on a commit, I'm adding this mechanism to the
>> configSet/conf. I'm not doing so for SolrCloud at this time because of
>> overhead concerns but it could be added later with some work.
>>
>> I also plan to remove QEC's support of the elevate.xml file to be
>> "versioned" -- e.g. elevate.xml.001, elevate.xml.002. The underlying
>> mechanism was added in
>> https://issues.apache.org/jira/browse/SOLR-351 where it was argued to be
>> useful for Windows OS which cannot replace an already open file. Obviously
>> this is a non-issue in SolrCloud, and perhaps not an issue if you use the
>> confFiles replication feature to edit it either. Regardless, I'm highly
>> suspicious it's an issue whatsoever since this file shouldn't be held open
>> on a sustained basis; it's only loaded on core load / replication and with
>> my changes above, on a commit if the modification time changes.
>>
>> If you have concerns, let me know.
>>
>> SOLR-15274 - QueryElevationComponent: auto-load file changes; remove data
>> dir support <https://issues.apache.org/jira/browse/SOLR-15274>
>>
>> ~ David Smiley
>> Apache Lucene/Solr Search Developer
>> http://www.linkedin.com/in/davidwsmiley
>>
>
>
>
> Disclaimer: This email and any files transmitted with it are confidential
> and intended solely for the use of the individual or entity to whom they are
> addressed. If you have received this email in error please notify the
> system manager. If you are not the named addressee you should not
> disseminate,
> distribute or copy this email. Please notify the sender immediately by
> email if you have received this email by mistake and delete this email from
> your
> system.
>
Re: Deprecation of QueryElevationComponent's support for elevate.xml
in data directories
Posted by Mónica Marrero <mo...@europeana.eu>.
It seems to be a cleaner way to manage this. Just a comment: what about a
mechanism to just update this file in memory? I know it will also involve
opening a new searcher, but at least It does not depend on commits in the
data (we may need to force changes in the data when what we really want to
do is to update the elevate.xml file).
On Sat, 20 Mar 2021 at 05:36, David Smiley <ds...@apache.org> wrote:
> Hey Solr community,
>
> I found myself doing some maintenance inside QueryElevationComponent
> related to some much larger refactoring. I'm seeing a fair amount of
> complexity around it's support of "elevate.xml" in a Solr *data* directory
> (where the index is). I don't know of any other Solr component that does
> anything like this. This feature of QEC existed from the start in Jan
> 2008. In October of that same year, Solr's replication handler gained the
> ability to replicate config files (see "confFiles" option) between cores.
> I think this aspect of QEC would not have existed if it came later. In
> SolrCloud, you're expected to use ZooKeeper of course.
>
> I plan to remove this mechanism in 9.0 on the grounds that it is obsolete.
>
> To account for the ability to automatically detect elevate.xml changes in
> a data dir and reload on a commit, I'm adding this mechanism to the
> configSet/conf. I'm not doing so for SolrCloud at this time because of
> overhead concerns but it could be added later with some work.
>
> I also plan to remove QEC's support of the elevate.xml file to be
> "versioned" -- e.g. elevate.xml.001, elevate.xml.002. The underlying
> mechanism was added in
> https://issues.apache.org/jira/browse/SOLR-351 where it was argued to be
> useful for Windows OS which cannot replace an already open file. Obviously
> this is a non-issue in SolrCloud, and perhaps not an issue if you use the
> confFiles replication feature to edit it either. Regardless, I'm highly
> suspicious it's an issue whatsoever since this file shouldn't be held open
> on a sustained basis; it's only loaded on core load / replication and with
> my changes above, on a commit if the modification time changes.
>
> If you have concerns, let me know.
>
> SOLR-15274 - QueryElevationComponent: auto-load file changes; remove data
> dir support <https://issues.apache.org/jira/browse/SOLR-15274>
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
--
Disclaimer: This email and any files transmitted with it are confidential
and intended solely for the use of the individual or entity to whom they
are
addressed. If you have received this email in error please notify the
system manager. If you are not the named addressee you should not
disseminate,
distribute or copy this email. Please notify the sender
immediately by email if you have received this email by mistake and delete
this email from your
system.