You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Vincenzo D'Amore <v....@gmail.com> on 2020/06/24 17:00:37 UTC
Developing update processor/Query Parser
Hi all,
I've started to work on a couple of components very tight together.
An update processor that writes few fields in the solr index and a Query
Parser that, well, then reads such fields from the index.
Such components share few configuration parameters together, I'm asking if
there is a pattern, a draft, a sample, some guidelines or best practices
that explains how to properly save configuration parameters.
The configuration is written into the solrconfig.xml file, for example:
<processor class="com.example.MyCustomProcessorFactory" >
<lst name="name">
<str name="n1">x1</str>
<str name="n2">x2</str>
</lst>
</processor>
And then query parser :
<queryParser name="myqp"
class="com.example.query.MyCustomQueryParserPlugin" />
I'm struggling because the change of configuration on the updated processor
has an impact on the query parser.
For example the configuration info shared between those two components can
be overwritten during a core reload.
Basically, during an update or a core reload, there is a query parser that
is serving requests while some other component is updating the index.
So I suppose there should be a pattern, an approach, a common solution when
a piece of configuration has to be loaded at boot, or when the core is
loaded.
Or when, after an update a new searcher is created and a new query parser
is created.
Any suggestion is really appreciated.
Best regards,
Vincenzo
--
Vincenzo D'Amore
Re: Developing update processor/Query Parser
Posted by Vincenzo D'Amore <v....@gmail.com>.
Sharing a static object between URP and QParser it's easy. But when the
core is reloaded, this shared static object is rewritten from scratch.
An old QParser that references that object could have serious problems,
inconsistencies, concurrent modification exceptions, etc.
On the other hand, trying to solve this problem using synchronization,
mutex or semaphores will lead to a non performing solution.
Another real problem I have, is that it is not clear what happens
internally.
On Fri, Jun 26, 2020 at 9:19 PM Vincenzo D'Amore <v....@gmail.com> wrote:
> Hi Gus, thanks for the thorough explanation.
> Infact, I was concerned about how to hold the shared information (between
> URP and QParser), for example when a core is reloaded.
> What happens when a core is reloaded? Very likely I have a couple of new
> URP/QParser but an older QParser may still be serving a request.
> If my assumption is true, now I have a clearer idea.
> So the question is, how to share an object (CustomConfig) between URP and
> QParser.
> And this CustomConfig object is created by URP every time the core is
> reloaded.
>
> On Fri, Jun 26, 2020 at 8:47 PM Gus Heck <gu...@gmail.com> wrote:
>
>> During the request, the parser plugin is retrieved from a PluginBag on the
>> SolrCore object, so it should be reloaded at the same time as the update
>> component (which comes from another PluginBag on SolrCore). If the
>> components are deployed with consistent configuration in solrconfig.xml,
>> any given SolrCore instance should have a consistent set of both. If you
>> want to avoid repeating the information, one possibility is to use a
>> system
>> property
>>
>> https://lucene.apache.org/solr/guide/8_4/configuring-solrconfig-xml.html#jvm-system-properties
>> though
>> the suitability of that may depend on the size of your cluster and your
>> deployment infrastructure.
>>
>> On Thu, Jun 25, 2020 at 2:47 PM Mikhail Khludnev <mk...@apache.org> wrote:
>>
>> > Hello, Vincenzo.
>> > Please find above about a dedicated component doing nothing, but just
>> > holding a config.
>> > Also you may extract config into a file and load it by
>> > SolrResourceLoaderAware.
>> >
>> > On Thu, Jun 25, 2020 at 2:06 PM Vincenzo D'Amore <v....@gmail.com>
>> > wrote:
>> >
>> > > Hi Mikhail, yup, I was trying to avoid putting logic in Solr.
>> > > Just to be a little bit more specific, consider that if the update
>> > factory
>> > > writes a field that has a size of 50.
>> > > The QParser should be aware of the current size when writing a query.
>> > >
>> > > Is it possible to have in solrconfig.xml file a shared configuration?
>> > >
>> > > I mean a snippet of configuration shared between update processor
>> factory
>> > > and QParser.
>> > >
>> > >
>> > > On Wed, Jun 24, 2020 at 10:33 PM Mikhail Khludnev <mk...@apache.org>
>> > wrote:
>> > >
>> > > > Hello, Vincenzo.
>> > > > Presumably you can introduce a component which just holds a config
>> > data,
>> > > > and then this component might be lookedup from QParser and
>> > UpdateFactory.
>> > > > Overall, it seems like embedding logic into Solr core, which rarely
>> > works
>> > > > well.
>> > > >
>> > > > On Wed, Jun 24, 2020 at 8:00 PM Vincenzo D'Amore <
>> v.damore@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > I've started to work on a couple of components very tight
>> together.
>> > > > > An update processor that writes few fields in the solr index and a
>> > > Query
>> > > > > Parser that, well, then reads such fields from the index.
>> > > > >
>> > > > > Such components share few configuration parameters together, I'm
>> > asking
>> > > > if
>> > > > > there is a pattern, a draft, a sample, some guidelines or best
>> > > practices
>> > > > > that explains how to properly save configuration parameters.
>> > > > >
>> > > > > The configuration is written into the solrconfig.xml file, for
>> > example:
>> > > > >
>> > > > > <processor class="com.example.MyCustomProcessorFactory" >
>> > > > > <lst name="name">
>> > > > > <str name="n1">x1</str>
>> > > > > <str name="n2">x2</str>
>> > > > > </lst>
>> > > > > </processor>
>> > > > >
>> > > > > And then query parser :
>> > > > >
>> > > > > <queryParser name="myqp"
>> > > > > class="com.example.query.MyCustomQueryParserPlugin" />
>> > > > >
>> > > > > I'm struggling because the change of configuration on the updated
>> > > > processor
>> > > > > has an impact on the query parser.
>> > > > > For example the configuration info shared between those two
>> > components
>> > > > can
>> > > > > be overwritten during a core reload.
>> > > > > Basically, during an update or a core reload, there is a query
>> parser
>> > > > that
>> > > > > is serving requests while some other component is updating the
>> index.
>> > > > > So I suppose there should be a pattern, an approach, a common
>> > solution
>> > > > when
>> > > > > a piece of configuration has to be loaded at boot, or when the
>> core
>> > is
>> > > > > loaded.
>> > > > > Or when, after an update a new searcher is created and a new query
>> > > parser
>> > > > > is created.
>> > > > >
>> > > > > Any suggestion is really appreciated.
>> > > > >
>> > > > > Best regards,
>> > > > > Vincenzo
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Vincenzo D'Amore
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Sincerely yours
>> > > > Mikhail Khludnev
>> > > >
>> > >
>> > >
>> > > --
>> > > Vincenzo D'Amore
>> > >
>> >
>> >
>> > --
>> > Sincerely yours
>> > Mikhail Khludnev
>> >
>>
>>
>> --
>> http://www.needhamsoftware.com (work)
>> http://www.the111shift.com (play)
>>
>
>
> --
> Vincenzo D'Amore
>
>
--
Vincenzo D'Amore
Re: Developing update processor/Query Parser
Posted by Vincenzo D'Amore <v....@gmail.com>.
Hi Gus, thanks for the thorough explanation.
Infact, I was concerned about how to hold the shared information (between
URP and QParser), for example when a core is reloaded.
What happens when a core is reloaded? Very likely I have a couple of new
URP/QParser but an older QParser may still be serving a request.
If my assumption is true, now I have a clearer idea.
So the question is, how to share an object (CustomConfig) between URP and
QParser.
And this CustomConfig object is created by URP every time the core is
reloaded.
On Fri, Jun 26, 2020 at 8:47 PM Gus Heck <gu...@gmail.com> wrote:
> During the request, the parser plugin is retrieved from a PluginBag on the
> SolrCore object, so it should be reloaded at the same time as the update
> component (which comes from another PluginBag on SolrCore). If the
> components are deployed with consistent configuration in solrconfig.xml,
> any given SolrCore instance should have a consistent set of both. If you
> want to avoid repeating the information, one possibility is to use a system
> property
>
> https://lucene.apache.org/solr/guide/8_4/configuring-solrconfig-xml.html#jvm-system-properties
> though
> the suitability of that may depend on the size of your cluster and your
> deployment infrastructure.
>
> On Thu, Jun 25, 2020 at 2:47 PM Mikhail Khludnev <mk...@apache.org> wrote:
>
> > Hello, Vincenzo.
> > Please find above about a dedicated component doing nothing, but just
> > holding a config.
> > Also you may extract config into a file and load it by
> > SolrResourceLoaderAware.
> >
> > On Thu, Jun 25, 2020 at 2:06 PM Vincenzo D'Amore <v....@gmail.com>
> > wrote:
> >
> > > Hi Mikhail, yup, I was trying to avoid putting logic in Solr.
> > > Just to be a little bit more specific, consider that if the update
> > factory
> > > writes a field that has a size of 50.
> > > The QParser should be aware of the current size when writing a query.
> > >
> > > Is it possible to have in solrconfig.xml file a shared configuration?
> > >
> > > I mean a snippet of configuration shared between update processor
> factory
> > > and QParser.
> > >
> > >
> > > On Wed, Jun 24, 2020 at 10:33 PM Mikhail Khludnev <mk...@apache.org>
> > wrote:
> > >
> > > > Hello, Vincenzo.
> > > > Presumably you can introduce a component which just holds a config
> > data,
> > > > and then this component might be lookedup from QParser and
> > UpdateFactory.
> > > > Overall, it seems like embedding logic into Solr core, which rarely
> > works
> > > > well.
> > > >
> > > > On Wed, Jun 24, 2020 at 8:00 PM Vincenzo D'Amore <v.damore@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I've started to work on a couple of components very tight together.
> > > > > An update processor that writes few fields in the solr index and a
> > > Query
> > > > > Parser that, well, then reads such fields from the index.
> > > > >
> > > > > Such components share few configuration parameters together, I'm
> > asking
> > > > if
> > > > > there is a pattern, a draft, a sample, some guidelines or best
> > > practices
> > > > > that explains how to properly save configuration parameters.
> > > > >
> > > > > The configuration is written into the solrconfig.xml file, for
> > example:
> > > > >
> > > > > <processor class="com.example.MyCustomProcessorFactory" >
> > > > > <lst name="name">
> > > > > <str name="n1">x1</str>
> > > > > <str name="n2">x2</str>
> > > > > </lst>
> > > > > </processor>
> > > > >
> > > > > And then query parser :
> > > > >
> > > > > <queryParser name="myqp"
> > > > > class="com.example.query.MyCustomQueryParserPlugin" />
> > > > >
> > > > > I'm struggling because the change of configuration on the updated
> > > > processor
> > > > > has an impact on the query parser.
> > > > > For example the configuration info shared between those two
> > components
> > > > can
> > > > > be overwritten during a core reload.
> > > > > Basically, during an update or a core reload, there is a query
> parser
> > > > that
> > > > > is serving requests while some other component is updating the
> index.
> > > > > So I suppose there should be a pattern, an approach, a common
> > solution
> > > > when
> > > > > a piece of configuration has to be loaded at boot, or when the core
> > is
> > > > > loaded.
> > > > > Or when, after an update a new searcher is created and a new query
> > > parser
> > > > > is created.
> > > > >
> > > > > Any suggestion is really appreciated.
> > > > >
> > > > > Best regards,
> > > > > Vincenzo
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Vincenzo D'Amore
> > > > >
> > > >
> > > >
> > > > --
> > > > Sincerely yours
> > > > Mikhail Khludnev
> > > >
> > >
> > >
> > > --
> > > Vincenzo D'Amore
> > >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> >
>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>
--
Vincenzo D'Amore
Re: Developing update processor/Query Parser
Posted by Gus Heck <gu...@gmail.com>.
During the request, the parser plugin is retrieved from a PluginBag on the
SolrCore object, so it should be reloaded at the same time as the update
component (which comes from another PluginBag on SolrCore). If the
components are deployed with consistent configuration in solrconfig.xml,
any given SolrCore instance should have a consistent set of both. If you
want to avoid repeating the information, one possibility is to use a system
property
https://lucene.apache.org/solr/guide/8_4/configuring-solrconfig-xml.html#jvm-system-properties
though
the suitability of that may depend on the size of your cluster and your
deployment infrastructure.
On Thu, Jun 25, 2020 at 2:47 PM Mikhail Khludnev <mk...@apache.org> wrote:
> Hello, Vincenzo.
> Please find above about a dedicated component doing nothing, but just
> holding a config.
> Also you may extract config into a file and load it by
> SolrResourceLoaderAware.
>
> On Thu, Jun 25, 2020 at 2:06 PM Vincenzo D'Amore <v....@gmail.com>
> wrote:
>
> > Hi Mikhail, yup, I was trying to avoid putting logic in Solr.
> > Just to be a little bit more specific, consider that if the update
> factory
> > writes a field that has a size of 50.
> > The QParser should be aware of the current size when writing a query.
> >
> > Is it possible to have in solrconfig.xml file a shared configuration?
> >
> > I mean a snippet of configuration shared between update processor factory
> > and QParser.
> >
> >
> > On Wed, Jun 24, 2020 at 10:33 PM Mikhail Khludnev <mk...@apache.org>
> wrote:
> >
> > > Hello, Vincenzo.
> > > Presumably you can introduce a component which just holds a config
> data,
> > > and then this component might be lookedup from QParser and
> UpdateFactory.
> > > Overall, it seems like embedding logic into Solr core, which rarely
> works
> > > well.
> > >
> > > On Wed, Jun 24, 2020 at 8:00 PM Vincenzo D'Amore <v....@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I've started to work on a couple of components very tight together.
> > > > An update processor that writes few fields in the solr index and a
> > Query
> > > > Parser that, well, then reads such fields from the index.
> > > >
> > > > Such components share few configuration parameters together, I'm
> asking
> > > if
> > > > there is a pattern, a draft, a sample, some guidelines or best
> > practices
> > > > that explains how to properly save configuration parameters.
> > > >
> > > > The configuration is written into the solrconfig.xml file, for
> example:
> > > >
> > > > <processor class="com.example.MyCustomProcessorFactory" >
> > > > <lst name="name">
> > > > <str name="n1">x1</str>
> > > > <str name="n2">x2</str>
> > > > </lst>
> > > > </processor>
> > > >
> > > > And then query parser :
> > > >
> > > > <queryParser name="myqp"
> > > > class="com.example.query.MyCustomQueryParserPlugin" />
> > > >
> > > > I'm struggling because the change of configuration on the updated
> > > processor
> > > > has an impact on the query parser.
> > > > For example the configuration info shared between those two
> components
> > > can
> > > > be overwritten during a core reload.
> > > > Basically, during an update or a core reload, there is a query parser
> > > that
> > > > is serving requests while some other component is updating the index.
> > > > So I suppose there should be a pattern, an approach, a common
> solution
> > > when
> > > > a piece of configuration has to be loaded at boot, or when the core
> is
> > > > loaded.
> > > > Or when, after an update a new searcher is created and a new query
> > parser
> > > > is created.
> > > >
> > > > Any suggestion is really appreciated.
> > > >
> > > > Best regards,
> > > > Vincenzo
> > > >
> > > >
> > > >
> > > > --
> > > > Vincenzo D'Amore
> > > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> > >
> >
> >
> > --
> > Vincenzo D'Amore
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>
--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)
Re: Developing update processor/Query Parser
Posted by Mikhail Khludnev <mk...@apache.org>.
Hello, Vincenzo.
Please find above about a dedicated component doing nothing, but just
holding a config.
Also you may extract config into a file and load it by
SolrResourceLoaderAware.
On Thu, Jun 25, 2020 at 2:06 PM Vincenzo D'Amore <v....@gmail.com> wrote:
> Hi Mikhail, yup, I was trying to avoid putting logic in Solr.
> Just to be a little bit more specific, consider that if the update factory
> writes a field that has a size of 50.
> The QParser should be aware of the current size when writing a query.
>
> Is it possible to have in solrconfig.xml file a shared configuration?
>
> I mean a snippet of configuration shared between update processor factory
> and QParser.
>
>
> On Wed, Jun 24, 2020 at 10:33 PM Mikhail Khludnev <mk...@apache.org> wrote:
>
> > Hello, Vincenzo.
> > Presumably you can introduce a component which just holds a config data,
> > and then this component might be lookedup from QParser and UpdateFactory.
> > Overall, it seems like embedding logic into Solr core, which rarely works
> > well.
> >
> > On Wed, Jun 24, 2020 at 8:00 PM Vincenzo D'Amore <v....@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I've started to work on a couple of components very tight together.
> > > An update processor that writes few fields in the solr index and a
> Query
> > > Parser that, well, then reads such fields from the index.
> > >
> > > Such components share few configuration parameters together, I'm asking
> > if
> > > there is a pattern, a draft, a sample, some guidelines or best
> practices
> > > that explains how to properly save configuration parameters.
> > >
> > > The configuration is written into the solrconfig.xml file, for example:
> > >
> > > <processor class="com.example.MyCustomProcessorFactory" >
> > > <lst name="name">
> > > <str name="n1">x1</str>
> > > <str name="n2">x2</str>
> > > </lst>
> > > </processor>
> > >
> > > And then query parser :
> > >
> > > <queryParser name="myqp"
> > > class="com.example.query.MyCustomQueryParserPlugin" />
> > >
> > > I'm struggling because the change of configuration on the updated
> > processor
> > > has an impact on the query parser.
> > > For example the configuration info shared between those two components
> > can
> > > be overwritten during a core reload.
> > > Basically, during an update or a core reload, there is a query parser
> > that
> > > is serving requests while some other component is updating the index.
> > > So I suppose there should be a pattern, an approach, a common solution
> > when
> > > a piece of configuration has to be loaded at boot, or when the core is
> > > loaded.
> > > Or when, after an update a new searcher is created and a new query
> parser
> > > is created.
> > >
> > > Any suggestion is really appreciated.
> > >
> > > Best regards,
> > > Vincenzo
> > >
> > >
> > >
> > > --
> > > Vincenzo D'Amore
> > >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> >
>
>
> --
> Vincenzo D'Amore
>
--
Sincerely yours
Mikhail Khludnev
Re: Developing update processor/Query Parser
Posted by Vincenzo D'Amore <v....@gmail.com>.
Hi Mikhail, yup, I was trying to avoid putting logic in Solr.
Just to be a little bit more specific, consider that if the update factory
writes a field that has a size of 50.
The QParser should be aware of the current size when writing a query.
Is it possible to have in solrconfig.xml file a shared configuration?
I mean a snippet of configuration shared between update processor factory
and QParser.
On Wed, Jun 24, 2020 at 10:33 PM Mikhail Khludnev <mk...@apache.org> wrote:
> Hello, Vincenzo.
> Presumably you can introduce a component which just holds a config data,
> and then this component might be lookedup from QParser and UpdateFactory.
> Overall, it seems like embedding logic into Solr core, which rarely works
> well.
>
> On Wed, Jun 24, 2020 at 8:00 PM Vincenzo D'Amore <v....@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I've started to work on a couple of components very tight together.
> > An update processor that writes few fields in the solr index and a Query
> > Parser that, well, then reads such fields from the index.
> >
> > Such components share few configuration parameters together, I'm asking
> if
> > there is a pattern, a draft, a sample, some guidelines or best practices
> > that explains how to properly save configuration parameters.
> >
> > The configuration is written into the solrconfig.xml file, for example:
> >
> > <processor class="com.example.MyCustomProcessorFactory" >
> > <lst name="name">
> > <str name="n1">x1</str>
> > <str name="n2">x2</str>
> > </lst>
> > </processor>
> >
> > And then query parser :
> >
> > <queryParser name="myqp"
> > class="com.example.query.MyCustomQueryParserPlugin" />
> >
> > I'm struggling because the change of configuration on the updated
> processor
> > has an impact on the query parser.
> > For example the configuration info shared between those two components
> can
> > be overwritten during a core reload.
> > Basically, during an update or a core reload, there is a query parser
> that
> > is serving requests while some other component is updating the index.
> > So I suppose there should be a pattern, an approach, a common solution
> when
> > a piece of configuration has to be loaded at boot, or when the core is
> > loaded.
> > Or when, after an update a new searcher is created and a new query parser
> > is created.
> >
> > Any suggestion is really appreciated.
> >
> > Best regards,
> > Vincenzo
> >
> >
> >
> > --
> > Vincenzo D'Amore
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>
--
Vincenzo D'Amore
Re: Developing update processor/Query Parser
Posted by Mikhail Khludnev <mk...@apache.org>.
Hello, Vincenzo.
Presumably you can introduce a component which just holds a config data,
and then this component might be lookedup from QParser and UpdateFactory.
Overall, it seems like embedding logic into Solr core, which rarely works
well.
On Wed, Jun 24, 2020 at 8:00 PM Vincenzo D'Amore <v....@gmail.com> wrote:
> Hi all,
>
> I've started to work on a couple of components very tight together.
> An update processor that writes few fields in the solr index and a Query
> Parser that, well, then reads such fields from the index.
>
> Such components share few configuration parameters together, I'm asking if
> there is a pattern, a draft, a sample, some guidelines or best practices
> that explains how to properly save configuration parameters.
>
> The configuration is written into the solrconfig.xml file, for example:
>
> <processor class="com.example.MyCustomProcessorFactory" >
> <lst name="name">
> <str name="n1">x1</str>
> <str name="n2">x2</str>
> </lst>
> </processor>
>
> And then query parser :
>
> <queryParser name="myqp"
> class="com.example.query.MyCustomQueryParserPlugin" />
>
> I'm struggling because the change of configuration on the updated processor
> has an impact on the query parser.
> For example the configuration info shared between those two components can
> be overwritten during a core reload.
> Basically, during an update or a core reload, there is a query parser that
> is serving requests while some other component is updating the index.
> So I suppose there should be a pattern, an approach, a common solution when
> a piece of configuration has to be loaded at boot, or when the core is
> loaded.
> Or when, after an update a new searcher is created and a new query parser
> is created.
>
> Any suggestion is really appreciated.
>
> Best regards,
> Vincenzo
>
>
>
> --
> Vincenzo D'Amore
>
--
Sincerely yours
Mikhail Khludnev