You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by gnandre <ar...@gmail.com> on 2022/11/14 06:33:31 UTC
How to write custom Solr plugin that can be called core container level instead of core level
e.g. /solr/admin/cores request works without specifying specific core in
the request.
I took a look at the code of such request handlers (e.g. LoggingHandler)
but they simply extend to RequestHandlerBase just like any other core-level
request handler. What makes them special so that they do not require core
context?
Re: How to write custom Solr plugin that can be called core container level instead of core level
Posted by gnandre <ar...@gmail.com>.
Thanks, Shawn. This is very helpful. I will use this solution!
On Sun, Nov 20, 2022 at 12:56 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 11/19/22 21:38, gnandre wrote:
> > Thanks, Shawn. I am a bit wary of creating a total core just for this
> > health check purpose.
> > Isn't it a bit expensive to create a core, considering the caches and so
> > many things it registers? Or, is it fine in which case I might just
> proceed
> > with that solution.
>
> I'm talking about a completely empty core that you never send queries
> to. If it never gets queries, it should not need any significant memory
> for caching.
>
> This is the absolute minimum config and schema I was able to use to make
> a new core like what you would need. There are no cache definitions, so
> the Solr caches should not even be created. It would probably still
> create some Lucene caches, but as those would be empty, they would be
> very small. With this definition I don't think you CAN send it any
> queries:
>
> solrconfig.xml:
> ---
> <?xml version="1.0" encoding="UTF-8" ?>
> <config>
> <luceneMatchVersion>LATEST</luceneMatchVersion>
> <requestHandler name="/your/handler/path" class="YOURCLASSNAME">
> *** ANY CONFIG NEEDED FOR YOUR HANDLER ***
> </requestHandler>
> </config>
> ---
>
> managed-schema.xml:
> ---
> <?xml version="1.0" encoding="UTF-8"?>
> <schema name="empty" version="1.6">
> <fieldType name="string" class="solr.StrField" omitNorms="true"
> sortMissingLast="true" docValues="true"/>
> <fieldType name="long" class="solr.LongPointField"
> positionIncrementGap="0" docValues="true"/>
> <field name="id" type="string" indexed="true" required="true"
> stored="true"/>
> <field name="_version_" type="long" indexed="false" stored="false"/>
> <uniqueKey>id</uniqueKey>
> </schema>
> ---
>
> This should result in the absolute minimum memory footprint for a core.
> Probably only a few megabytes. And I think there is some work underway
> to try and reduce the memory footprint for all cores in some future
> version of Solr. That would in theory make the memory requirement for
> this core even smaller.
>
> In the future I hope that we can provide a way for users to register
> global handlers and make this empty core nonsense unnecessary.
>
> Thanks,
> Shawn
>
>
Re: How to write custom Solr plugin that can be called core container level instead of core level
Posted by Shawn Heisey <ap...@elyograg.org>.
On 11/19/22 21:38, gnandre wrote:
> Thanks, Shawn. I am a bit wary of creating a total core just for this
> health check purpose.
> Isn't it a bit expensive to create a core, considering the caches and so
> many things it registers? Or, is it fine in which case I might just proceed
> with that solution.
I'm talking about a completely empty core that you never send queries
to. If it never gets queries, it should not need any significant memory
for caching.
This is the absolute minimum config and schema I was able to use to make
a new core like what you would need. There are no cache definitions, so
the Solr caches should not even be created. It would probably still
create some Lucene caches, but as those would be empty, they would be
very small. With this definition I don't think you CAN send it any queries:
solrconfig.xml:
---
<?xml version="1.0" encoding="UTF-8" ?>
<config>
<luceneMatchVersion>LATEST</luceneMatchVersion>
<requestHandler name="/your/handler/path" class="YOURCLASSNAME">
*** ANY CONFIG NEEDED FOR YOUR HANDLER ***
</requestHandler>
</config>
---
managed-schema.xml:
---
<?xml version="1.0" encoding="UTF-8"?>
<schema name="empty" version="1.6">
<fieldType name="string" class="solr.StrField" omitNorms="true"
sortMissingLast="true" docValues="true"/>
<fieldType name="long" class="solr.LongPointField"
positionIncrementGap="0" docValues="true"/>
<field name="id" type="string" indexed="true" required="true"
stored="true"/>
<field name="_version_" type="long" indexed="false" stored="false"/>
<uniqueKey>id</uniqueKey>
</schema>
---
This should result in the absolute minimum memory footprint for a core.
Probably only a few megabytes. And I think there is some work underway
to try and reduce the memory footprint for all cores in some future
version of Solr. That would in theory make the memory requirement for
this core even smaller.
In the future I hope that we can provide a way for users to register
global handlers and make this empty core nonsense unnecessary.
Thanks,
Shawn
Re: How to write custom Solr plugin that can be called core container level instead of core level
Posted by gnandre <ar...@gmail.com>.
Thanks, Shawn. I am a bit wary of creating a total core just for this
health check purpose.
Isn't it a bit expensive to create a core, considering the caches and so
many things it registers? Or, is it fine in which case I might just proceed
with that solution.
On Sat, Nov 19, 2022 at 6:00 AM Shawn Heisey <ap...@elyograg.org> wrote:
> On 11/17/22 22:05, gnandre wrote:
> > Unfortunately, I am still on legacy mode and not cloud mode.
> > So, I can't take advantage of this feature :(
> >
> > In the meantime, I have implemented the Solr plugin and it is working as
> > expected but the only issue is that it needs to be called in the
> > context of
> > some core.
> > I wonder if we can reference cores with numbers? e.g. /solr/0/my/api/call
> > instead of /solr/<some_core_name>/my/api/call where - refers to first
> core
> > in some sense (say alphabetical)
> > That would alleviate the issue at least a bit because I don't need to
> know
> > the name of any particular core to get some node level info.
>
> Sending again to the list. Accidentally sent only to end user. Twice!
>
> Currently, global handlers (/admin/collections and /admin/cores for
> example) are added by code in the CoreContainer class.
>
> Here's an idea for fellow devs to consider: Can we have those handlers
> specified instead by a file similar to ImplicitPlugins.json, maybe
> NodeImplicitPlugins.json or CoreContainerImplicitPlugins.json that would
> live in the solr-core jar along with ImplicitPlugins.json? And have
> both code paths look for User* varieties of those filenames on the
> classpath? Then users could just add UserImplicitPlugins.json and/or
> UserNodeImplicitPlugins.json to ${solr.solr.home}/lib along with the
> user-supplied jar(s) that implement them and those handlers would be
> automatically available.
>
> This is not to say that your idea of numbering cores is wrong ... just
> that if your plugin is for the node or cluster, it shouldn't be accessed
> at the core/collection level, and then there is no need for core
> numbering. One thing you could do until something like the idea above
> is added is create an empty core with a specific name on every node with
> the handler in solrconfig.xml and then use that core name to access your
> handler.
>
> Thanks,
> Shawn
>
>
Re: How to write custom Solr plugin that can be called core container level instead of core level
Posted by Shawn Heisey <ap...@elyograg.org>.
On 11/17/22 22:05, gnandre wrote:
> Unfortunately, I am still on legacy mode and not cloud mode.
> So, I can't take advantage of this feature :(
>
> In the meantime, I have implemented the Solr plugin and it is working as
> expected but the only issue is that it needs to be called in the
> context of
> some core.
> I wonder if we can reference cores with numbers? e.g. /solr/0/my/api/call
> instead of /solr/<some_core_name>/my/api/call where - refers to first core
> in some sense (say alphabetical)
> That would alleviate the issue at least a bit because I don't need to know
> the name of any particular core to get some node level info.
Sending again to the list. Accidentally sent only to end user. Twice!
Currently, global handlers (/admin/collections and /admin/cores for
example) are added by code in the CoreContainer class.
Here's an idea for fellow devs to consider: Can we have those handlers
specified instead by a file similar to ImplicitPlugins.json, maybe
NodeImplicitPlugins.json or CoreContainerImplicitPlugins.json that would
live in the solr-core jar along with ImplicitPlugins.json? And have
both code paths look for User* varieties of those filenames on the
classpath? Then users could just add UserImplicitPlugins.json and/or
UserNodeImplicitPlugins.json to ${solr.solr.home}/lib along with the
user-supplied jar(s) that implement them and those handlers would be
automatically available.
This is not to say that your idea of numbering cores is wrong ... just
that if your plugin is for the node or cluster, it shouldn't be accessed
at the core/collection level, and then there is no need for core
numbering. One thing you could do until something like the idea above
is added is create an empty core with a specific name on every node with
the handler in solrconfig.xml and then use that core name to access your
handler.
Thanks,
Shawn
Re: How to write custom Solr plugin that can be called core container level instead of core level
Posted by gnandre <ar...@gmail.com>.
Unfortunately, I am still on legacy mode and not cloud mode.
So, I can't take advantage of this feature :(
In the meantime, I have implemented the Solr plugin and it is working as
expected but the only issue is that it needs to be called in the context of
some core.
I wonder if we can reference cores with numbers? e.g. /solr/0/my/api/call
instead of /solr/<some_core_name>/my/api/call where - refers to first core
in some sense (say alphabetical)
That would alleviate the issue at least a bit because I don't need to know
the name of any particular core to get some node level info.
On Wed, Nov 16, 2022 at 7:03 AM Charlie Hull <
chull@opensourceconnections.com> wrote:
> Once you figure this out it would be awesome if you could write up what
> you did - a blog would be a fantastic resource for people writing
> plugins in the future.
>
> Best
>
> Charlie
>
> On 16/11/2022 11:52, Jan Høydahl wrote:
> > Yes
> >
> > Seehttps://
> solr.apache.org/guide/solr/latest/configuration-guide/cluster-plugins.html
> for the docs.
> >
> > Jan
> >
> >> 16. nov. 2022 kl. 00:43 skrev gnandre<ar...@gmail.com>:
> >>
> >> Also, is this supported only in SolrCloud mode?
> >>
> >> On Tue, Nov 15, 2022 at 6:42 PM gnandre<ar...@gmail.com>
> wrote:
> >>
> >>> Thanks, Ishan. Is this available only in Solr 8.6+? I am using 8.5.0 :(
> >>>
> >>> On Tue, Nov 15, 2022 at 2:19 PM Ishan Chattopadhyaya <
> >>> ichattopadhyaya@gmail.com> wrote:
> >>>
> >>>> https://issues.apache.org/jira/browse/SOLR-14404
> >>>>
> >>>> On Wed, Nov 16, 2022 at 12:46 AM Ishan Chattopadhyaya <
> >>>> ichattopadhyaya@gmail.com> wrote:
> >>>>
> >>>>> Here's a core container level plugin, for example:
> >>>>> https://github.com/yasa-org/yasa/tree/master/yasa-solr-plugin
> >>>>>
> >>>>> On Mon, Nov 14, 2022 at 10:07 PM Gus Heck<gu...@gmail.com>
> wrote:
> >>>>>
> >>>>>> In 9.x it should be possible to write a separate servlet that can
> >>>> answer
> >>>>>> custom non-search queries. Then all you need to edit is web.xml. You
> >>>> can
> >>>>>> now get hold of a core container
> >>>>>> via org.apache.solr.servlet.CoreContainerProvider#getCoreContainer.
> >>>>>>
> >>>>>> Looking at the code, it seems you still need to live in the apache
> >>>> package
> >>>>>> because that method is not public, but if you've got a demonstrable
> use
> >>>>>> case for it I don't see why that couldn't be made public in future
> >>>>>> versions. If that's tweaked to be public you could avoid changing
> >>>> implicit
> >>>>>> plugins, would not need to live under admin, and it should be
> possible
> >>>> to
> >>>>>> load from a separate jar if the jar is inserted
> >>>>>> into /server/solr-webapp/webapp/WEB-INF/lib
> >>>>>>
> >>>>>> A question to consider when choosing is whether you want the end
> point
> >>>> to
> >>>>>> be protected by authentication. If so then you're going to want to
> do
> >>>> it
> >>>>>> Shawn's way. If you want something publicly visible (say to
> >>>> infrastructure
> >>>>>> systems without login) then a servlet might be more useful since it
> >>>> will
> >>>>>> live outside of all our security stuff.
> >>>>>>
> >>>>>> On Mon, Nov 14, 2022 at 2:16 AM gnandre<ar...@gmail.com>
> >>>> wrote:
> >>>>>>> Thanks, Shawn. I think what you propose there will be very helpful.
> >>>>>> There
> >>>>>>> are definitely usecases where we want to work at the solr node
> level
> >>>> and
> >>>>>>> not core level. Fieldcache is one other example.
> >>>>>>>
> >>>>>>> In my case, I am trying to write a simple health check request
> >>>> handler
> >>>>>> that
> >>>>>>> makes sure that all cores on a particular node are loaded and are
> >>>>>>> queriable. The ping request handler works on a core level. The
> >>>>>>> /solr/admin/cores api works for me but it returns with the ok
> status
> >>>>>> even
> >>>>>>> if some other cores are still loading after solr restart.
> >>>>>>>
> >>>>>>> Ps. I am using legacy setup (non-cloud)
> >>>>>>>
> >>>>>>> On Mon, Nov 14, 2022, 2:04 AM Shawn Heisey<ap...@elyograg.org>
> >>>> wrote:
> >>>>>>>> On 11/13/22 23:55, Shawn Heisey wrote:
> >>>>>>>>> If you want to create your own global handler, then you can add a
> >>>>>>>>> definition to ImplicitPlugins.json, which is embedded in the
> >>>>>> solr-core
> >>>>>>>>> jar. Unless you want to do jar surgery, it's best to make that
> >>>>>> change
> >>>>>>>>> in the source code and recompile Solr.
> >>>>>>>> Seehttps://issues.apache.org/jira/browse/SOLR-15859 for an
> >>>> example
> >>>>>>>> where I am creating a new global handler.
> >>>>>>>>
> >>>>>>>> I am wondering if ImplicitPlugins.json could be put in a location
> >>>> like
> >>>>>>>> server/lib/ext so recompiling Solr or jar surgery isn't required.
> >>>> I
> >>>>>>>> think server/lib/ext is searched on the classpath before WEB-INF.
> >>>>>> Even
> >>>>>>>> better would be to have another json where user-specified implicit
> >>>>>>>> handlers can be defined.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Shawn
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> http://www.needhamsoftware.com (work)
> >>>>>> http://www.the111shift.com (play)
> >>>>>>
> >
> --
> Charlie Hull - Managing Consultant at OpenSource Connections Limited
> Founding member of The Search Network <http://www.thesearchnetwork.com>
> and co-author of Searching the Enterprise
> <
> https://opensourceconnections.com/wp-content/uploads/2020/08/ES_book_final_journal_version.pdf
> >
> tel/fax: +44 (0)8700 118334
> mobile: +44 (0)7767 825828
>
> OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
> Amtsgericht Charlottenburg | HRB 230712 B
> Geschäftsführer: John M. Woodell | David E. Pugh
> Finanzamt: Berlin Finanzamt für Körperschaften II
Re: How to write custom Solr plugin that can be called core container level instead of core level
Posted by Charlie Hull <ch...@opensourceconnections.com>.
Once you figure this out it would be awesome if you could write up what
you did - a blog would be a fantastic resource for people writing
plugins in the future.
Best
Charlie
On 16/11/2022 11:52, Jan Høydahl wrote:
> Yes
>
> Seehttps://solr.apache.org/guide/solr/latest/configuration-guide/cluster-plugins.html for the docs.
>
> Jan
>
>> 16. nov. 2022 kl. 00:43 skrev gnandre<ar...@gmail.com>:
>>
>> Also, is this supported only in SolrCloud mode?
>>
>> On Tue, Nov 15, 2022 at 6:42 PM gnandre<ar...@gmail.com> wrote:
>>
>>> Thanks, Ishan. Is this available only in Solr 8.6+? I am using 8.5.0 :(
>>>
>>> On Tue, Nov 15, 2022 at 2:19 PM Ishan Chattopadhyaya <
>>> ichattopadhyaya@gmail.com> wrote:
>>>
>>>> https://issues.apache.org/jira/browse/SOLR-14404
>>>>
>>>> On Wed, Nov 16, 2022 at 12:46 AM Ishan Chattopadhyaya <
>>>> ichattopadhyaya@gmail.com> wrote:
>>>>
>>>>> Here's a core container level plugin, for example:
>>>>> https://github.com/yasa-org/yasa/tree/master/yasa-solr-plugin
>>>>>
>>>>> On Mon, Nov 14, 2022 at 10:07 PM Gus Heck<gu...@gmail.com> wrote:
>>>>>
>>>>>> In 9.x it should be possible to write a separate servlet that can
>>>> answer
>>>>>> custom non-search queries. Then all you need to edit is web.xml. You
>>>> can
>>>>>> now get hold of a core container
>>>>>> via org.apache.solr.servlet.CoreContainerProvider#getCoreContainer.
>>>>>>
>>>>>> Looking at the code, it seems you still need to live in the apache
>>>> package
>>>>>> because that method is not public, but if you've got a demonstrable use
>>>>>> case for it I don't see why that couldn't be made public in future
>>>>>> versions. If that's tweaked to be public you could avoid changing
>>>> implicit
>>>>>> plugins, would not need to live under admin, and it should be possible
>>>> to
>>>>>> load from a separate jar if the jar is inserted
>>>>>> into /server/solr-webapp/webapp/WEB-INF/lib
>>>>>>
>>>>>> A question to consider when choosing is whether you want the end point
>>>> to
>>>>>> be protected by authentication. If so then you're going to want to do
>>>> it
>>>>>> Shawn's way. If you want something publicly visible (say to
>>>> infrastructure
>>>>>> systems without login) then a servlet might be more useful since it
>>>> will
>>>>>> live outside of all our security stuff.
>>>>>>
>>>>>> On Mon, Nov 14, 2022 at 2:16 AM gnandre<ar...@gmail.com>
>>>> wrote:
>>>>>>> Thanks, Shawn. I think what you propose there will be very helpful.
>>>>>> There
>>>>>>> are definitely usecases where we want to work at the solr node level
>>>> and
>>>>>>> not core level. Fieldcache is one other example.
>>>>>>>
>>>>>>> In my case, I am trying to write a simple health check request
>>>> handler
>>>>>> that
>>>>>>> makes sure that all cores on a particular node are loaded and are
>>>>>>> queriable. The ping request handler works on a core level. The
>>>>>>> /solr/admin/cores api works for me but it returns with the ok status
>>>>>> even
>>>>>>> if some other cores are still loading after solr restart.
>>>>>>>
>>>>>>> Ps. I am using legacy setup (non-cloud)
>>>>>>>
>>>>>>> On Mon, Nov 14, 2022, 2:04 AM Shawn Heisey<ap...@elyograg.org>
>>>> wrote:
>>>>>>>> On 11/13/22 23:55, Shawn Heisey wrote:
>>>>>>>>> If you want to create your own global handler, then you can add a
>>>>>>>>> definition to ImplicitPlugins.json, which is embedded in the
>>>>>> solr-core
>>>>>>>>> jar. Unless you want to do jar surgery, it's best to make that
>>>>>> change
>>>>>>>>> in the source code and recompile Solr.
>>>>>>>> Seehttps://issues.apache.org/jira/browse/SOLR-15859 for an
>>>> example
>>>>>>>> where I am creating a new global handler.
>>>>>>>>
>>>>>>>> I am wondering if ImplicitPlugins.json could be put in a location
>>>> like
>>>>>>>> server/lib/ext so recompiling Solr or jar surgery isn't required.
>>>> I
>>>>>>>> think server/lib/ext is searched on the classpath before WEB-INF.
>>>>>> Even
>>>>>>>> better would be to have another json where user-specified implicit
>>>>>>>> handlers can be defined.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Shawn
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> http://www.needhamsoftware.com (work)
>>>>>> http://www.the111shift.com (play)
>>>>>>
>
--
Charlie Hull - Managing Consultant at OpenSource Connections Limited
Founding member of The Search Network <http://www.thesearchnetwork.com>
and co-author of Searching the Enterprise
<https://opensourceconnections.com/wp-content/uploads/2020/08/ES_book_final_journal_version.pdf>
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
OpenSource Connections Europe GmbH | Pappelallee 78/79 | 10437 Berlin
Amtsgericht Charlottenburg | HRB 230712 B
Geschäftsführer: John M. Woodell | David E. Pugh
Finanzamt: Berlin Finanzamt für Körperschaften II
Re: How to write custom Solr plugin that can be called core container level instead of core level
Posted by Jan Høydahl <ja...@cominvent.com>.
Yes
See https://solr.apache.org/guide/solr/latest/configuration-guide/cluster-plugins.html for the docs.
Jan
> 16. nov. 2022 kl. 00:43 skrev gnandre <ar...@gmail.com>:
>
> Also, is this supported only in SolrCloud mode?
>
> On Tue, Nov 15, 2022 at 6:42 PM gnandre <ar...@gmail.com> wrote:
>
>> Thanks, Ishan. Is this available only in Solr 8.6+? I am using 8.5.0 :(
>>
>> On Tue, Nov 15, 2022 at 2:19 PM Ishan Chattopadhyaya <
>> ichattopadhyaya@gmail.com> wrote:
>>
>>> https://issues.apache.org/jira/browse/SOLR-14404
>>>
>>> On Wed, Nov 16, 2022 at 12:46 AM Ishan Chattopadhyaya <
>>> ichattopadhyaya@gmail.com> wrote:
>>>
>>>> Here's a core container level plugin, for example:
>>>> https://github.com/yasa-org/yasa/tree/master/yasa-solr-plugin
>>>>
>>>> On Mon, Nov 14, 2022 at 10:07 PM Gus Heck <gu...@gmail.com> wrote:
>>>>
>>>>> In 9.x it should be possible to write a separate servlet that can
>>> answer
>>>>> custom non-search queries. Then all you need to edit is web.xml. You
>>> can
>>>>> now get hold of a core container
>>>>> via org.apache.solr.servlet.CoreContainerProvider#getCoreContainer.
>>>>>
>>>>> Looking at the code, it seems you still need to live in the apache
>>> package
>>>>> because that method is not public, but if you've got a demonstrable use
>>>>> case for it I don't see why that couldn't be made public in future
>>>>> versions. If that's tweaked to be public you could avoid changing
>>> implicit
>>>>> plugins, would not need to live under admin, and it should be possible
>>> to
>>>>> load from a separate jar if the jar is inserted
>>>>> into /server/solr-webapp/webapp/WEB-INF/lib
>>>>>
>>>>> A question to consider when choosing is whether you want the end point
>>> to
>>>>> be protected by authentication. If so then you're going to want to do
>>> it
>>>>> Shawn's way. If you want something publicly visible (say to
>>> infrastructure
>>>>> systems without login) then a servlet might be more useful since it
>>> will
>>>>> live outside of all our security stuff.
>>>>>
>>>>> On Mon, Nov 14, 2022 at 2:16 AM gnandre <ar...@gmail.com>
>>> wrote:
>>>>>
>>>>>> Thanks, Shawn. I think what you propose there will be very helpful.
>>>>> There
>>>>>> are definitely usecases where we want to work at the solr node level
>>> and
>>>>>> not core level. Fieldcache is one other example.
>>>>>>
>>>>>> In my case, I am trying to write a simple health check request
>>> handler
>>>>> that
>>>>>> makes sure that all cores on a particular node are loaded and are
>>>>>> queriable. The ping request handler works on a core level. The
>>>>>> /solr/admin/cores api works for me but it returns with the ok status
>>>>> even
>>>>>> if some other cores are still loading after solr restart.
>>>>>>
>>>>>> Ps. I am using legacy setup (non-cloud)
>>>>>>
>>>>>> On Mon, Nov 14, 2022, 2:04 AM Shawn Heisey <ap...@elyograg.org>
>>> wrote:
>>>>>>
>>>>>>> On 11/13/22 23:55, Shawn Heisey wrote:
>>>>>>>> If you want to create your own global handler, then you can add a
>>>>>>>> definition to ImplicitPlugins.json, which is embedded in the
>>>>> solr-core
>>>>>>>> jar. Unless you want to do jar surgery, it's best to make that
>>>>> change
>>>>>>>> in the source code and recompile Solr.
>>>>>>>
>>>>>>> See https://issues.apache.org/jira/browse/SOLR-15859 for an
>>> example
>>>>>>> where I am creating a new global handler.
>>>>>>>
>>>>>>> I am wondering if ImplicitPlugins.json could be put in a location
>>> like
>>>>>>> server/lib/ext so recompiling Solr or jar surgery isn't required.
>>> I
>>>>>>> think server/lib/ext is searched on the classpath before WEB-INF.
>>>>> Even
>>>>>>> better would be to have another json where user-specified implicit
>>>>>>> handlers can be defined.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Shawn
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> http://www.needhamsoftware.com (work)
>>>>> http://www.the111shift.com (play)
>>>>>
>>>>
>>>
>>
Re: How to write custom Solr plugin that can be called core container level instead of core level
Posted by gnandre <ar...@gmail.com>.
Also, is this supported only in SolrCloud mode?
On Tue, Nov 15, 2022 at 6:42 PM gnandre <ar...@gmail.com> wrote:
> Thanks, Ishan. Is this available only in Solr 8.6+? I am using 8.5.0 :(
>
> On Tue, Nov 15, 2022 at 2:19 PM Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
>
>> https://issues.apache.org/jira/browse/SOLR-14404
>>
>> On Wed, Nov 16, 2022 at 12:46 AM Ishan Chattopadhyaya <
>> ichattopadhyaya@gmail.com> wrote:
>>
>> > Here's a core container level plugin, for example:
>> > https://github.com/yasa-org/yasa/tree/master/yasa-solr-plugin
>> >
>> > On Mon, Nov 14, 2022 at 10:07 PM Gus Heck <gu...@gmail.com> wrote:
>> >
>> >> In 9.x it should be possible to write a separate servlet that can
>> answer
>> >> custom non-search queries. Then all you need to edit is web.xml. You
>> can
>> >> now get hold of a core container
>> >> via org.apache.solr.servlet.CoreContainerProvider#getCoreContainer.
>> >>
>> >> Looking at the code, it seems you still need to live in the apache
>> package
>> >> because that method is not public, but if you've got a demonstrable use
>> >> case for it I don't see why that couldn't be made public in future
>> >> versions. If that's tweaked to be public you could avoid changing
>> implicit
>> >> plugins, would not need to live under admin, and it should be possible
>> to
>> >> load from a separate jar if the jar is inserted
>> >> into /server/solr-webapp/webapp/WEB-INF/lib
>> >>
>> >> A question to consider when choosing is whether you want the end point
>> to
>> >> be protected by authentication. If so then you're going to want to do
>> it
>> >> Shawn's way. If you want something publicly visible (say to
>> infrastructure
>> >> systems without login) then a servlet might be more useful since it
>> will
>> >> live outside of all our security stuff.
>> >>
>> >> On Mon, Nov 14, 2022 at 2:16 AM gnandre <ar...@gmail.com>
>> wrote:
>> >>
>> >> > Thanks, Shawn. I think what you propose there will be very helpful.
>> >> There
>> >> > are definitely usecases where we want to work at the solr node level
>> and
>> >> > not core level. Fieldcache is one other example.
>> >> >
>> >> > In my case, I am trying to write a simple health check request
>> handler
>> >> that
>> >> > makes sure that all cores on a particular node are loaded and are
>> >> > queriable. The ping request handler works on a core level. The
>> >> > /solr/admin/cores api works for me but it returns with the ok status
>> >> even
>> >> > if some other cores are still loading after solr restart.
>> >> >
>> >> > Ps. I am using legacy setup (non-cloud)
>> >> >
>> >> > On Mon, Nov 14, 2022, 2:04 AM Shawn Heisey <ap...@elyograg.org>
>> wrote:
>> >> >
>> >> > > On 11/13/22 23:55, Shawn Heisey wrote:
>> >> > > > If you want to create your own global handler, then you can add a
>> >> > > > definition to ImplicitPlugins.json, which is embedded in the
>> >> solr-core
>> >> > > > jar. Unless you want to do jar surgery, it's best to make that
>> >> change
>> >> > > > in the source code and recompile Solr.
>> >> > >
>> >> > > See https://issues.apache.org/jira/browse/SOLR-15859 for an
>> example
>> >> > > where I am creating a new global handler.
>> >> > >
>> >> > > I am wondering if ImplicitPlugins.json could be put in a location
>> like
>> >> > > server/lib/ext so recompiling Solr or jar surgery isn't required.
>> I
>> >> > > think server/lib/ext is searched on the classpath before WEB-INF.
>> >> Even
>> >> > > better would be to have another json where user-specified implicit
>> >> > > handlers can be defined.
>> >> > >
>> >> > > Thanks,
>> >> > > Shawn
>> >> > >
>> >> > >
>> >> >
>> >>
>> >>
>> >> --
>> >> http://www.needhamsoftware.com (work)
>> >> http://www.the111shift.com (play)
>> >>
>> >
>>
>
Re: How to write custom Solr plugin that can be called core container level instead of core level
Posted by gnandre <ar...@gmail.com>.
Thanks, Ishan. Is this available only in Solr 8.6+? I am using 8.5.0 :(
On Tue, Nov 15, 2022 at 2:19 PM Ishan Chattopadhyaya <
ichattopadhyaya@gmail.com> wrote:
> https://issues.apache.org/jira/browse/SOLR-14404
>
> On Wed, Nov 16, 2022 at 12:46 AM Ishan Chattopadhyaya <
> ichattopadhyaya@gmail.com> wrote:
>
> > Here's a core container level plugin, for example:
> > https://github.com/yasa-org/yasa/tree/master/yasa-solr-plugin
> >
> > On Mon, Nov 14, 2022 at 10:07 PM Gus Heck <gu...@gmail.com> wrote:
> >
> >> In 9.x it should be possible to write a separate servlet that can answer
> >> custom non-search queries. Then all you need to edit is web.xml. You can
> >> now get hold of a core container
> >> via org.apache.solr.servlet.CoreContainerProvider#getCoreContainer.
> >>
> >> Looking at the code, it seems you still need to live in the apache
> package
> >> because that method is not public, but if you've got a demonstrable use
> >> case for it I don't see why that couldn't be made public in future
> >> versions. If that's tweaked to be public you could avoid changing
> implicit
> >> plugins, would not need to live under admin, and it should be possible
> to
> >> load from a separate jar if the jar is inserted
> >> into /server/solr-webapp/webapp/WEB-INF/lib
> >>
> >> A question to consider when choosing is whether you want the end point
> to
> >> be protected by authentication. If so then you're going to want to do it
> >> Shawn's way. If you want something publicly visible (say to
> infrastructure
> >> systems without login) then a servlet might be more useful since it will
> >> live outside of all our security stuff.
> >>
> >> On Mon, Nov 14, 2022 at 2:16 AM gnandre <ar...@gmail.com>
> wrote:
> >>
> >> > Thanks, Shawn. I think what you propose there will be very helpful.
> >> There
> >> > are definitely usecases where we want to work at the solr node level
> and
> >> > not core level. Fieldcache is one other example.
> >> >
> >> > In my case, I am trying to write a simple health check request handler
> >> that
> >> > makes sure that all cores on a particular node are loaded and are
> >> > queriable. The ping request handler works on a core level. The
> >> > /solr/admin/cores api works for me but it returns with the ok status
> >> even
> >> > if some other cores are still loading after solr restart.
> >> >
> >> > Ps. I am using legacy setup (non-cloud)
> >> >
> >> > On Mon, Nov 14, 2022, 2:04 AM Shawn Heisey <ap...@elyograg.org>
> wrote:
> >> >
> >> > > On 11/13/22 23:55, Shawn Heisey wrote:
> >> > > > If you want to create your own global handler, then you can add a
> >> > > > definition to ImplicitPlugins.json, which is embedded in the
> >> solr-core
> >> > > > jar. Unless you want to do jar surgery, it's best to make that
> >> change
> >> > > > in the source code and recompile Solr.
> >> > >
> >> > > See https://issues.apache.org/jira/browse/SOLR-15859 for an example
> >> > > where I am creating a new global handler.
> >> > >
> >> > > I am wondering if ImplicitPlugins.json could be put in a location
> like
> >> > > server/lib/ext so recompiling Solr or jar surgery isn't required. I
> >> > > think server/lib/ext is searched on the classpath before WEB-INF.
> >> Even
> >> > > better would be to have another json where user-specified implicit
> >> > > handlers can be defined.
> >> > >
> >> > > Thanks,
> >> > > Shawn
> >> > >
> >> > >
> >> >
> >>
> >>
> >> --
> >> http://www.needhamsoftware.com (work)
> >> http://www.the111shift.com (play)
> >>
> >
>
Re: How to write custom Solr plugin that can be called core container level instead of core level
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
https://issues.apache.org/jira/browse/SOLR-14404
On Wed, Nov 16, 2022 at 12:46 AM Ishan Chattopadhyaya <
ichattopadhyaya@gmail.com> wrote:
> Here's a core container level plugin, for example:
> https://github.com/yasa-org/yasa/tree/master/yasa-solr-plugin
>
> On Mon, Nov 14, 2022 at 10:07 PM Gus Heck <gu...@gmail.com> wrote:
>
>> In 9.x it should be possible to write a separate servlet that can answer
>> custom non-search queries. Then all you need to edit is web.xml. You can
>> now get hold of a core container
>> via org.apache.solr.servlet.CoreContainerProvider#getCoreContainer.
>>
>> Looking at the code, it seems you still need to live in the apache package
>> because that method is not public, but if you've got a demonstrable use
>> case for it I don't see why that couldn't be made public in future
>> versions. If that's tweaked to be public you could avoid changing implicit
>> plugins, would not need to live under admin, and it should be possible to
>> load from a separate jar if the jar is inserted
>> into /server/solr-webapp/webapp/WEB-INF/lib
>>
>> A question to consider when choosing is whether you want the end point to
>> be protected by authentication. If so then you're going to want to do it
>> Shawn's way. If you want something publicly visible (say to infrastructure
>> systems without login) then a servlet might be more useful since it will
>> live outside of all our security stuff.
>>
>> On Mon, Nov 14, 2022 at 2:16 AM gnandre <ar...@gmail.com> wrote:
>>
>> > Thanks, Shawn. I think what you propose there will be very helpful.
>> There
>> > are definitely usecases where we want to work at the solr node level and
>> > not core level. Fieldcache is one other example.
>> >
>> > In my case, I am trying to write a simple health check request handler
>> that
>> > makes sure that all cores on a particular node are loaded and are
>> > queriable. The ping request handler works on a core level. The
>> > /solr/admin/cores api works for me but it returns with the ok status
>> even
>> > if some other cores are still loading after solr restart.
>> >
>> > Ps. I am using legacy setup (non-cloud)
>> >
>> > On Mon, Nov 14, 2022, 2:04 AM Shawn Heisey <ap...@elyograg.org> wrote:
>> >
>> > > On 11/13/22 23:55, Shawn Heisey wrote:
>> > > > If you want to create your own global handler, then you can add a
>> > > > definition to ImplicitPlugins.json, which is embedded in the
>> solr-core
>> > > > jar. Unless you want to do jar surgery, it's best to make that
>> change
>> > > > in the source code and recompile Solr.
>> > >
>> > > See https://issues.apache.org/jira/browse/SOLR-15859 for an example
>> > > where I am creating a new global handler.
>> > >
>> > > I am wondering if ImplicitPlugins.json could be put in a location like
>> > > server/lib/ext so recompiling Solr or jar surgery isn't required. I
>> > > think server/lib/ext is searched on the classpath before WEB-INF.
>> Even
>> > > better would be to have another json where user-specified implicit
>> > > handlers can be defined.
>> > >
>> > > Thanks,
>> > > Shawn
>> > >
>> > >
>> >
>>
>>
>> --
>> http://www.needhamsoftware.com (work)
>> http://www.the111shift.com (play)
>>
>
Re: How to write custom Solr plugin that can be called core container level instead of core level
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
Here's a core container level plugin, for example:
https://github.com/yasa-org/yasa/tree/master/yasa-solr-plugin
On Mon, Nov 14, 2022 at 10:07 PM Gus Heck <gu...@gmail.com> wrote:
> In 9.x it should be possible to write a separate servlet that can answer
> custom non-search queries. Then all you need to edit is web.xml. You can
> now get hold of a core container
> via org.apache.solr.servlet.CoreContainerProvider#getCoreContainer.
>
> Looking at the code, it seems you still need to live in the apache package
> because that method is not public, but if you've got a demonstrable use
> case for it I don't see why that couldn't be made public in future
> versions. If that's tweaked to be public you could avoid changing implicit
> plugins, would not need to live under admin, and it should be possible to
> load from a separate jar if the jar is inserted
> into /server/solr-webapp/webapp/WEB-INF/lib
>
> A question to consider when choosing is whether you want the end point to
> be protected by authentication. If so then you're going to want to do it
> Shawn's way. If you want something publicly visible (say to infrastructure
> systems without login) then a servlet might be more useful since it will
> live outside of all our security stuff.
>
> On Mon, Nov 14, 2022 at 2:16 AM gnandre <ar...@gmail.com> wrote:
>
> > Thanks, Shawn. I think what you propose there will be very helpful. There
> > are definitely usecases where we want to work at the solr node level and
> > not core level. Fieldcache is one other example.
> >
> > In my case, I am trying to write a simple health check request handler
> that
> > makes sure that all cores on a particular node are loaded and are
> > queriable. The ping request handler works on a core level. The
> > /solr/admin/cores api works for me but it returns with the ok status even
> > if some other cores are still loading after solr restart.
> >
> > Ps. I am using legacy setup (non-cloud)
> >
> > On Mon, Nov 14, 2022, 2:04 AM Shawn Heisey <ap...@elyograg.org> wrote:
> >
> > > On 11/13/22 23:55, Shawn Heisey wrote:
> > > > If you want to create your own global handler, then you can add a
> > > > definition to ImplicitPlugins.json, which is embedded in the
> solr-core
> > > > jar. Unless you want to do jar surgery, it's best to make that
> change
> > > > in the source code and recompile Solr.
> > >
> > > See https://issues.apache.org/jira/browse/SOLR-15859 for an example
> > > where I am creating a new global handler.
> > >
> > > I am wondering if ImplicitPlugins.json could be put in a location like
> > > server/lib/ext so recompiling Solr or jar surgery isn't required. I
> > > think server/lib/ext is searched on the classpath before WEB-INF. Even
> > > better would be to have another json where user-specified implicit
> > > handlers can be defined.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>
Re: How to write custom Solr plugin that can be called core container level instead of core level
Posted by Gus Heck <gu...@gmail.com>.
In 9.x it should be possible to write a separate servlet that can answer
custom non-search queries. Then all you need to edit is web.xml. You can
now get hold of a core container
via org.apache.solr.servlet.CoreContainerProvider#getCoreContainer.
Looking at the code, it seems you still need to live in the apache package
because that method is not public, but if you've got a demonstrable use
case for it I don't see why that couldn't be made public in future
versions. If that's tweaked to be public you could avoid changing implicit
plugins, would not need to live under admin, and it should be possible to
load from a separate jar if the jar is inserted
into /server/solr-webapp/webapp/WEB-INF/lib
A question to consider when choosing is whether you want the end point to
be protected by authentication. If so then you're going to want to do it
Shawn's way. If you want something publicly visible (say to infrastructure
systems without login) then a servlet might be more useful since it will
live outside of all our security stuff.
On Mon, Nov 14, 2022 at 2:16 AM gnandre <ar...@gmail.com> wrote:
> Thanks, Shawn. I think what you propose there will be very helpful. There
> are definitely usecases where we want to work at the solr node level and
> not core level. Fieldcache is one other example.
>
> In my case, I am trying to write a simple health check request handler that
> makes sure that all cores on a particular node are loaded and are
> queriable. The ping request handler works on a core level. The
> /solr/admin/cores api works for me but it returns with the ok status even
> if some other cores are still loading after solr restart.
>
> Ps. I am using legacy setup (non-cloud)
>
> On Mon, Nov 14, 2022, 2:04 AM Shawn Heisey <ap...@elyograg.org> wrote:
>
> > On 11/13/22 23:55, Shawn Heisey wrote:
> > > If you want to create your own global handler, then you can add a
> > > definition to ImplicitPlugins.json, which is embedded in the solr-core
> > > jar. Unless you want to do jar surgery, it's best to make that change
> > > in the source code and recompile Solr.
> >
> > See https://issues.apache.org/jira/browse/SOLR-15859 for an example
> > where I am creating a new global handler.
> >
> > I am wondering if ImplicitPlugins.json could be put in a location like
> > server/lib/ext so recompiling Solr or jar surgery isn't required. I
> > think server/lib/ext is searched on the classpath before WEB-INF. Even
> > better would be to have another json where user-specified implicit
> > handlers can be defined.
> >
> > Thanks,
> > Shawn
> >
> >
>
--
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)
Re: How to write custom Solr plugin that can be called core container level instead of core level
Posted by gnandre <ar...@gmail.com>.
Thanks, Shawn. I think what you propose there will be very helpful. There
are definitely usecases where we want to work at the solr node level and
not core level. Fieldcache is one other example.
In my case, I am trying to write a simple health check request handler that
makes sure that all cores on a particular node are loaded and are
queriable. The ping request handler works on a core level. The
/solr/admin/cores api works for me but it returns with the ok status even
if some other cores are still loading after solr restart.
Ps. I am using legacy setup (non-cloud)
On Mon, Nov 14, 2022, 2:04 AM Shawn Heisey <ap...@elyograg.org> wrote:
> On 11/13/22 23:55, Shawn Heisey wrote:
> > If you want to create your own global handler, then you can add a
> > definition to ImplicitPlugins.json, which is embedded in the solr-core
> > jar. Unless you want to do jar surgery, it's best to make that change
> > in the source code and recompile Solr.
>
> See https://issues.apache.org/jira/browse/SOLR-15859 for an example
> where I am creating a new global handler.
>
> I am wondering if ImplicitPlugins.json could be put in a location like
> server/lib/ext so recompiling Solr or jar surgery isn't required. I
> think server/lib/ext is searched on the classpath before WEB-INF. Even
> better would be to have another json where user-specified implicit
> handlers can be defined.
>
> Thanks,
> Shawn
>
>
Re: How to write custom Solr plugin that can be called core container level instead of core level
Posted by Shawn Heisey <ap...@elyograg.org>.
On 11/13/22 23:55, Shawn Heisey wrote:
> If you want to create your own global handler, then you can add a
> definition to ImplicitPlugins.json, which is embedded in the solr-core
> jar. Unless you want to do jar surgery, it's best to make that change
> in the source code and recompile Solr.
See https://issues.apache.org/jira/browse/SOLR-15859 for an example
where I am creating a new global handler.
I am wondering if ImplicitPlugins.json could be put in a location like
server/lib/ext so recompiling Solr or jar surgery isn't required. I
think server/lib/ext is searched on the classpath before WEB-INF. Even
better would be to have another json where user-specified implicit
handlers can be defined.
Thanks,
Shawn
Re: How to write custom Solr plugin that can be called core container level instead of core level
Posted by Shawn Heisey <ap...@elyograg.org>.
On 11/13/22 23:33, gnandre wrote:
> e.g. /solr/admin/cores request works without specifying specific core in
> the request.
>
> I took a look at the code of such request handlers (e.g. LoggingHandler)
> but they simply extend to RequestHandlerBase just like any other core-level
> request handler. What makes them special so that they do not require core
> context?
The /solr/admin/cores handler, as well as any other handler starting
with /solr/admin and possibly a few others as well are *global*
handlers. If you want to create your own global handler, then you can
add a definition to ImplicitPlugins.json, which is embedded in the
solr-core jar. Unless you want to do jar surgery, it's best to make
that change in the source code and recompile Solr. I think that you can
add the jar for the handler to server/lib/ext but there are situations
(which I can't say whether yours would qualify) where the jar needs to
be added to WEB-INF/lib in the webapp.
Many handlers that get created are specific to each core, and so they
start with /solr/CORENAME and will only operate on the named core. You
should never create a core named admin, as it will interfere with the
global handlers.
Note that if you are in cloud mode (running Solr with ZooKeeper), then
you should never use /solr/admin/cores to do anything other than reload
a core. Any actual changes must be done with /solr/admin/collections or
you run the risk of really screwing up your cluster. It is recommended
to use the collections handler for reloads too.
Thanks,
Shawn