You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by jimtronic <ji...@gmail.com> on 2016/09/06 20:48:18 UTC
Can't create collection without plugin, can't load plugin without
collection
I've run into an orchestration problem while creating collections and loading
plugins via the ConfigAPI in Solr Cloud.
Here's the scenario:
1. I create a configSet that references a custom class in schema.xml.
2. I upload the jar to the BlobStore and issue add-runtimelib using the
Config API. This fails because the collection doesn't exist yet.
3. I try to create the collection with the configSet but it fails because
the custom plugin is not available yet.
I can force this to work by removing the custom reference, create the
collection, load the jar and then add the custom reference back in place.
This is fine as a manual one-time setup, but not feasible in a scripted
production deployment.
I wish I could create a collection without actually needing to create any
cores. Then, I could get all of the configurations for a collection setup
before creating the cores.
--
View this message in context: http://lucene.472066.n3.nabble.com/Can-t-create-collection-without-plugin-can-t-load-plugin-without-collection-tp4294865.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Can't create collection without plugin, can't load plugin without
collection
Posted by Chris Hostetter <ho...@fucit.org>.
Isn't the designed workflow for something like this...
1) upload a very minimal configset
2) create your collection
3) upload your plugin jar
4) use the Schema API to create the new fieldtype that uses your plugin
?
: Date: Wed, 7 Sep 2016 11:07:39 -0600
: From: John Bickerstaff <jo...@johnbickerstaff.com>
: Reply-To: solr-user@lucene.apache.org
: To: solr-user@lucene.apache.org
: Subject: Re: Can't create collection without plugin,
: can't load plugin without collection
:
: I might be on the wrong track, but this statement made me think...
:
: I can force this to work by removing the custom reference, create the
: collection, load the jar and then add the custom reference back in place.
: This is fine as a manual one-time setup, but not feasible in a scripted
: production deployment.
:
: If you're in a SolrCloud setup, it's only one step to change ALL the
: configs, right? Just upload "improved" configs to zookeeper from one box,
: restart all boxes. The improved configs could already reside on one (or
: all) of your SolrCloud servers (vms or physical machines - or be on a
: shared network drive)
:
: Unless I'm missing something (and I could be) it's fine for the jar to be
: in the appropriate folder on each box as long as it's NOT referenced in the
: config file... So that could be part of your build for every machine (or
: you could use a script to deploy it)
:
: Just trying to help think around the corners of the box - feel free to
: ignore if I've missed something obvious.
:
:
: On Wed, Sep 7, 2016 at 10:44 AM, Erick Erickson <er...@gmail.com>
: wrote:
:
: > Well, it was worth a try ;)....
: >
: > On Wed, Sep 7, 2016 at 9:42 AM, John Bickerstaff
: > <jo...@johnbickerstaff.com> wrote:
: > > Not sure if it's helpful, but I did solve a similar problem by scripting
: > > the placement of the plugin .jar file into the appropriate directories as
: > > part of the build process for my VMs
: > >
: > > I get that doing it through the BlobStore would be preferred, probably...
: > > But I couldn't get it to work and decided to go with "ye olde school"
: > bash
: > > scripting which I knew I could depend on.
: > >
: > > (Caveat - I have a fairly simple environment, so your mileage may vary)
: > >
: > > On Wed, Sep 7, 2016 at 10:38 AM, jimtronic <ji...@gmail.com> wrote:
: > >
: > >> Sadly, that didn't work.
: > >>
: > >> Without a core to hit, the /[COLLECTION]/config returns a 404 error.
: > >>
: > >> The best bet at this point may be for me may be one of the following:
: > >>
: > >> 1. Programmatically modify configoverlay.json file to add the runtime
: > libs
: > >> when I upload the config.
: > >>
: > >> or
: > >>
: > >> 2. Patch solr so that schema.xml loads custom classes directly from the
: > >> BlobStore like solrconfig.xml does.
: > >>
: > >> or
: > >>
: > >> 3. Patch solr so that you can specify configSets instead of a collection
: > >> when associating a runtimeLib.
: > >>
: > >>
: > >>
: > >>
: > >>
: > >>
: > >> --
: > >> View this message in context: http://lucene.472066.n3.
: > >> nabble.com/Can-t-create-collection-without-plugin-can-
: > >> t-load-plugin-without-collection-tp4294865p4295028.html
: > >> Sent from the Solr - User mailing list archive at Nabble.com.
: > >>
: >
:
-Hoss
http://www.lucidworks.com/
Re: Can't create collection without plugin, can't load plugin without collection
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
I might be on the wrong track, but this statement made me think...
I can force this to work by removing the custom reference, create the
collection, load the jar and then add the custom reference back in place.
This is fine as a manual one-time setup, but not feasible in a scripted
production deployment.
If you're in a SolrCloud setup, it's only one step to change ALL the
configs, right? Just upload "improved" configs to zookeeper from one box,
restart all boxes. The improved configs could already reside on one (or
all) of your SolrCloud servers (vms or physical machines - or be on a
shared network drive)
Unless I'm missing something (and I could be) it's fine for the jar to be
in the appropriate folder on each box as long as it's NOT referenced in the
config file... So that could be part of your build for every machine (or
you could use a script to deploy it)
Just trying to help think around the corners of the box - feel free to
ignore if I've missed something obvious.
On Wed, Sep 7, 2016 at 10:44 AM, Erick Erickson <er...@gmail.com>
wrote:
> Well, it was worth a try ;)....
>
> On Wed, Sep 7, 2016 at 9:42 AM, John Bickerstaff
> <jo...@johnbickerstaff.com> wrote:
> > Not sure if it's helpful, but I did solve a similar problem by scripting
> > the placement of the plugin .jar file into the appropriate directories as
> > part of the build process for my VMs
> >
> > I get that doing it through the BlobStore would be preferred, probably...
> > But I couldn't get it to work and decided to go with "ye olde school"
> bash
> > scripting which I knew I could depend on.
> >
> > (Caveat - I have a fairly simple environment, so your mileage may vary)
> >
> > On Wed, Sep 7, 2016 at 10:38 AM, jimtronic <ji...@gmail.com> wrote:
> >
> >> Sadly, that didn't work.
> >>
> >> Without a core to hit, the /[COLLECTION]/config returns a 404 error.
> >>
> >> The best bet at this point may be for me may be one of the following:
> >>
> >> 1. Programmatically modify configoverlay.json file to add the runtime
> libs
> >> when I upload the config.
> >>
> >> or
> >>
> >> 2. Patch solr so that schema.xml loads custom classes directly from the
> >> BlobStore like solrconfig.xml does.
> >>
> >> or
> >>
> >> 3. Patch solr so that you can specify configSets instead of a collection
> >> when associating a runtimeLib.
> >>
> >>
> >>
> >>
> >>
> >>
> >> --
> >> View this message in context: http://lucene.472066.n3.
> >> nabble.com/Can-t-create-collection-without-plugin-can-
> >> t-load-plugin-without-collection-tp4294865p4295028.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
>
Re: Can't create collection without plugin, can't load plugin without collection
Posted by Erick Erickson <er...@gmail.com>.
Well, it was worth a try ;)....
On Wed, Sep 7, 2016 at 9:42 AM, John Bickerstaff
<jo...@johnbickerstaff.com> wrote:
> Not sure if it's helpful, but I did solve a similar problem by scripting
> the placement of the plugin .jar file into the appropriate directories as
> part of the build process for my VMs
>
> I get that doing it through the BlobStore would be preferred, probably...
> But I couldn't get it to work and decided to go with "ye olde school" bash
> scripting which I knew I could depend on.
>
> (Caveat - I have a fairly simple environment, so your mileage may vary)
>
> On Wed, Sep 7, 2016 at 10:38 AM, jimtronic <ji...@gmail.com> wrote:
>
>> Sadly, that didn't work.
>>
>> Without a core to hit, the /[COLLECTION]/config returns a 404 error.
>>
>> The best bet at this point may be for me may be one of the following:
>>
>> 1. Programmatically modify configoverlay.json file to add the runtime libs
>> when I upload the config.
>>
>> or
>>
>> 2. Patch solr so that schema.xml loads custom classes directly from the
>> BlobStore like solrconfig.xml does.
>>
>> or
>>
>> 3. Patch solr so that you can specify configSets instead of a collection
>> when associating a runtimeLib.
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://lucene.472066.n3.
>> nabble.com/Can-t-create-collection-without-plugin-can-
>> t-load-plugin-without-collection-tp4294865p4295028.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
Re: Can't create collection without plugin, can't load plugin without collection
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
Not sure if it's helpful, but I did solve a similar problem by scripting
the placement of the plugin .jar file into the appropriate directories as
part of the build process for my VMs
I get that doing it through the BlobStore would be preferred, probably...
But I couldn't get it to work and decided to go with "ye olde school" bash
scripting which I knew I could depend on.
(Caveat - I have a fairly simple environment, so your mileage may vary)
On Wed, Sep 7, 2016 at 10:38 AM, jimtronic <ji...@gmail.com> wrote:
> Sadly, that didn't work.
>
> Without a core to hit, the /[COLLECTION]/config returns a 404 error.
>
> The best bet at this point may be for me may be one of the following:
>
> 1. Programmatically modify configoverlay.json file to add the runtime libs
> when I upload the config.
>
> or
>
> 2. Patch solr so that schema.xml loads custom classes directly from the
> BlobStore like solrconfig.xml does.
>
> or
>
> 3. Patch solr so that you can specify configSets instead of a collection
> when associating a runtimeLib.
>
>
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/Can-t-create-collection-without-plugin-can-
> t-load-plugin-without-collection-tp4294865p4295028.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Can't create collection without plugin, can't load plugin
without collection
Posted by jimtronic <ji...@gmail.com>.
Sadly, that didn't work.
Without a core to hit, the /[COLLECTION]/config returns a 404 error.
The best bet at this point may be for me may be one of the following:
1. Programmatically modify configoverlay.json file to add the runtime libs
when I upload the config.
or
2. Patch solr so that schema.xml loads custom classes directly from the
BlobStore like solrconfig.xml does.
or
3. Patch solr so that you can specify configSets instead of a collection
when associating a runtimeLib.
--
View this message in context: http://lucene.472066.n3.nabble.com/Can-t-create-collection-without-plugin-can-t-load-plugin-without-collection-tp4294865p4295028.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Can't create collection without plugin, can't load plugin without collection
Posted by Erick Erickson <er...@gmail.com>.
Have you tried creating the collection with the special value of
"EMPTY" for createNodeSet? See:
https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api1
It's original purpose was to allow _all_ replicas to be placed
manually, but it might work for your use-case.
WARNING: I haven't tried this in your exact scenario, so.....
Best,
Erick
On Tue, Sep 6, 2016 at 1:48 PM, jimtronic <ji...@gmail.com> wrote:
> I've run into an orchestration problem while creating collections and loading
> plugins via the ConfigAPI in Solr Cloud.
>
> Here's the scenario:
>
> 1. I create a configSet that references a custom class in schema.xml.
> 2. I upload the jar to the BlobStore and issue add-runtimelib using the
> Config API. This fails because the collection doesn't exist yet.
> 3. I try to create the collection with the configSet but it fails because
> the custom plugin is not available yet.
>
> I can force this to work by removing the custom reference, create the
> collection, load the jar and then add the custom reference back in place.
> This is fine as a manual one-time setup, but not feasible in a scripted
> production deployment.
>
> I wish I could create a collection without actually needing to create any
> cores. Then, I could get all of the configurations for a collection setup
> before creating the cores.
>
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Can-t-create-collection-without-plugin-can-t-load-plugin-without-collection-tp4294865.html
> Sent from the Solr - User mailing list archive at Nabble.com.