You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Don Bosco Durai <bo...@apache.org> on 2016/04/06 21:21:50 UTC
Adding configset in SolrCloud via API
Is there an equivalent of server/scripts/cloud-scripts/zkcli.sh -zkhost $zk_host -cmd upconfig -confdir $config_folder -confname $config_name using APIs?
I want to bootstrap by uploading the configs via API. Once the configs are uploaded, I am now able to do everything else via API.
Thanks
Bosco
RE: Adding configset in SolrCloud via API
Posted by "Davis, Daniel (NIH/NLM) [C]" <da...@nih.gov>.
Anshum,
The project https://github.com/danizen/solr-config-tool has code that uploads a configset to ZooKeeper via API. It depends only on SolrJ.
You'll want to look at the following Java classes under src/main/java:
org.danizen.solrconfig.SolrConfig
org.danizen.solrconfig.tests.CanUpConfig
The idea here was to build a command-line that front-ends some JUnit test cases that can be used to do a sort of "Smoke Test" of a Solr Configuration directory.
It isn't at all ready for prime time - it can create a collection but doesn't do any indexing or querying. I could elaborate on my thoughts, but it would be a bit technical about how to verify that indexing "worked". This is part of why the command-line tool only acts as a front-end for JUnit tests. I'd imagined that some users of this (which isn't ready for prime time), would build their own test plans that include some of these unit tests and some of their own.
The goal was to make Continuous Integration in my organization (and yours) a little easier - rather than looking at a lot of command-line output, just look at unit test results formatted by whatever CI tool you use.
________________________________________
From: Erick Erickson [erickerickson@gmail.com]
Sent: Wednesday, April 06, 2016 5:17 PM
To: solr-user
Subject: Re: Adding configset in SolrCloud via API
As of Solr 5.5 the bin/solr script can do this, see:
https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference
It's still not quite what you're looking for, but uploading arbitrary
xml scripts through a browser is a security issue, so it's possible
there will never be an API call to do that.
Best
Erick
On Wed, Apr 6, 2016 at 1:52 PM, Anshum Gupta <an...@anshumgupta.net> wrote:
> As of now, there's no way to do so. There were some efforts on those lines but it's been on hold.
>
> -Anshum
>
>> On Apr 6, 2016, at 12:21 PM, Don Bosco Durai <bo...@apache.org> wrote:
>>
>> Is there an equivalent of server/scripts/cloud-scripts/zkcli.sh -zkhost $zk_host -cmd upconfig -confdir $config_folder -confname $config_name using APIs?
>>
>> I want to bootstrap by uploading the configs via API. Once the configs are uploaded, I am now able to do everything else via API.
>>
>> Thanks
>>
>> Bosco
>>
>>
Re: Adding configset in SolrCloud via API
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
Hmmm...Not sure I understand, but it sounds like you've found the best
solution for the limitations you're experiencing...
On Wed, Apr 6, 2016 at 4:38 PM, Don Bosco Durai <bo...@apache.org> wrote:
> My challenge is, the server where my application is running doesn’t have
> Solr bits installed.
>
>
>
>
> Right now I am asking users to install (just unzip) solr on any server and
> I give them a shell script to run the script from command line before
> starting my application. It is inconvenient, so I was seeing if anyone was
> able to automate it.
>
> Thanks
>
> Bosco
>
>
>
>
> On 4/6/16, 2:47 PM, "John Bickerstaff" <jo...@johnbickerstaff.com> wrote:
>
> >Therefore, this becomes possible:
> >
> http://stackoverflow.com/questions/525212/how-to-run-unix-shell-script-from-java-code
> >
> >Hackish, but certainly doable... Given there's no API...
> >
> >On Wed, Apr 6, 2016 at 3:44 PM, John Bickerstaff <
> john@johnbickerstaff.com>
> >wrote:
> >
> >> Yup - just tested - that command runs fine with Solr NOT running...
> >>
> >> On Wed, Apr 6, 2016 at 3:41 PM, John Bickerstaff <
> john@johnbickerstaff.com
> >> > wrote:
> >>
> >>> If you can get to the IP addresses from your application, then there's
> >>> probably a way... Do you mean you're firewalled off or in some other
> way
> >>> unable to access the Solr box IP's from your Java application?
> >>>
> >>> If you're looking to do "automated build of virtual machines" there are
> >>> some tools like Vagrant...
> >>>
> >>> https://en.wikipedia.org/wiki/Vagrant_(software)
> >>>
> >>> Also, you probably don't need to be directly on one of the 'official"
> >>> SOLR machines to load the configs. I haven't tested, but as long as
> you
> >>> have the configs and a VM or server running SOLR, you could do this
> >>> yourself.
> >>>
> >>> The following command (as far as I've been able to tell) ONLY creates
> the
> >>> conf "directory" in Zookeeper... nothing else...
> >>>
> >>> And, as a matter of fact, I'm almost positive SOLR did not need to be
> >>> running when I did this, but I did so many variations while trying to
> >>> figure out how to bring up a new 5.4 collection that I'm not positive
> - I
> >>> could be totally wrong on that one.
> >>>
> >>> My point is that I uploaded the configs from a machine that had no
> >>> collection yet... it worked fine to set up then configs in Zookeeper.
> >>> Later, I issued the create collection command and referenced the
> config in
> >>> Zookeeper with the -n(?) flag...
> >>>
> >>> sudo /opt/solr/server/scripts/cloud-scripts/zkcli.sh -cmd upconfig
> >>> -confdir /home/john/conf/ -confname statdx -z 192.168.56.5/solr5_4
> >>>
> >>> On Wed, Apr 6, 2016 at 3:26 PM, Don Bosco Durai <bo...@apache.org>
> wrote:
> >>>
> >>>> I have SolrCloud pre-installed. I need to create a collection, but
> >>>> before that I need to load the config into zookeeper.
> >>>>
> >>>> I want to automate the entire process from my Java process which is
> not
> >>>> running on any of the servers were SolrCloud is running. In short, I
> don’t
> >>>> have access to bin/solr or server/scripts/cloud-scripts, etc from my
> >>>> application. So I was wondering if there were any way, like uploading
> a zip
> >>>> with the configs (schema.xml, solrconfig.xml, etc.). One workaround I
> can
> >>>> thinking is of making direct zookeeper calls.
> >>>>
> >>>> Anshum, thanks for your reply. I will see if I can find the JIRA.
> >>>>
> >>>> Thanks
> >>>>
> >>>>
> >>>> Bosco
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 4/6/16, 2:17 PM, "Erick Erickson" <er...@gmail.com> wrote:
> >>>>
> >>>> >As of Solr 5.5 the bin/solr script can do this, see:
> >>>> >
> >>>>
> https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference
> >>>> >
> >>>> >It's still not quite what you're looking for, but uploading arbitrary
> >>>> >xml scripts through a browser is a security issue, so it's possible
> >>>> >there will never be an API call to do that.
> >>>> >
> >>>> >Best
> >>>> >Erick
> >>>> >
> >>>> >On Wed, Apr 6, 2016 at 1:52 PM, Anshum Gupta <anshum@anshumgupta.net
> >
> >>>> wrote:
> >>>> >> As of now, there's no way to do so. There were some efforts on
> those
> >>>> lines but it's been on hold.
> >>>> >>
> >>>> >> -Anshum
> >>>> >>
> >>>> >>> On Apr 6, 2016, at 12:21 PM, Don Bosco Durai <bo...@apache.org>
> >>>> wrote:
> >>>> >>>
> >>>> >>> Is there an equivalent of server/scripts/cloud-scripts/zkcli.sh
> >>>> -zkhost $zk_host -cmd upconfig -confdir $config_folder -confname
> >>>> $config_name using APIs?
> >>>> >>>
> >>>> >>> I want to bootstrap by uploading the configs via API. Once the
> >>>> configs are uploaded, I am now able to do everything else via API.
> >>>> >>>
> >>>> >>> Thanks
> >>>> >>>
> >>>> >>> Bosco
> >>>> >>>
> >>>> >>>
> >>>>
> >>>>
> >>>
> >>
>
>
Re: Adding configset in SolrCloud via API
Posted by Don Bosco Durai <bo...@apache.org>.
My challenge is, the server where my application is running doesn’t have Solr bits installed.
Right now I am asking users to install (just unzip) solr on any server and I give them a shell script to run the script from command line before starting my application. It is inconvenient, so I was seeing if anyone was able to automate it.
Thanks
Bosco
On 4/6/16, 2:47 PM, "John Bickerstaff" <jo...@johnbickerstaff.com> wrote:
>Therefore, this becomes possible:
>http://stackoverflow.com/questions/525212/how-to-run-unix-shell-script-from-java-code
>
>Hackish, but certainly doable... Given there's no API...
>
>On Wed, Apr 6, 2016 at 3:44 PM, John Bickerstaff <jo...@johnbickerstaff.com>
>wrote:
>
>> Yup - just tested - that command runs fine with Solr NOT running...
>>
>> On Wed, Apr 6, 2016 at 3:41 PM, John Bickerstaff <john@johnbickerstaff.com
>> > wrote:
>>
>>> If you can get to the IP addresses from your application, then there's
>>> probably a way... Do you mean you're firewalled off or in some other way
>>> unable to access the Solr box IP's from your Java application?
>>>
>>> If you're looking to do "automated build of virtual machines" there are
>>> some tools like Vagrant...
>>>
>>> https://en.wikipedia.org/wiki/Vagrant_(software)
>>>
>>> Also, you probably don't need to be directly on one of the 'official"
>>> SOLR machines to load the configs. I haven't tested, but as long as you
>>> have the configs and a VM or server running SOLR, you could do this
>>> yourself.
>>>
>>> The following command (as far as I've been able to tell) ONLY creates the
>>> conf "directory" in Zookeeper... nothing else...
>>>
>>> And, as a matter of fact, I'm almost positive SOLR did not need to be
>>> running when I did this, but I did so many variations while trying to
>>> figure out how to bring up a new 5.4 collection that I'm not positive - I
>>> could be totally wrong on that one.
>>>
>>> My point is that I uploaded the configs from a machine that had no
>>> collection yet... it worked fine to set up then configs in Zookeeper.
>>> Later, I issued the create collection command and referenced the config in
>>> Zookeeper with the -n(?) flag...
>>>
>>> sudo /opt/solr/server/scripts/cloud-scripts/zkcli.sh -cmd upconfig
>>> -confdir /home/john/conf/ -confname statdx -z 192.168.56.5/solr5_4
>>>
>>> On Wed, Apr 6, 2016 at 3:26 PM, Don Bosco Durai <bo...@apache.org> wrote:
>>>
>>>> I have SolrCloud pre-installed. I need to create a collection, but
>>>> before that I need to load the config into zookeeper.
>>>>
>>>> I want to automate the entire process from my Java process which is not
>>>> running on any of the servers were SolrCloud is running. In short, I don’t
>>>> have access to bin/solr or server/scripts/cloud-scripts, etc from my
>>>> application. So I was wondering if there were any way, like uploading a zip
>>>> with the configs (schema.xml, solrconfig.xml, etc.). One workaround I can
>>>> thinking is of making direct zookeeper calls.
>>>>
>>>> Anshum, thanks for your reply. I will see if I can find the JIRA.
>>>>
>>>> Thanks
>>>>
>>>>
>>>> Bosco
>>>>
>>>>
>>>>
>>>>
>>>> On 4/6/16, 2:17 PM, "Erick Erickson" <er...@gmail.com> wrote:
>>>>
>>>> >As of Solr 5.5 the bin/solr script can do this, see:
>>>> >
>>>> https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference
>>>> >
>>>> >It's still not quite what you're looking for, but uploading arbitrary
>>>> >xml scripts through a browser is a security issue, so it's possible
>>>> >there will never be an API call to do that.
>>>> >
>>>> >Best
>>>> >Erick
>>>> >
>>>> >On Wed, Apr 6, 2016 at 1:52 PM, Anshum Gupta <an...@anshumgupta.net>
>>>> wrote:
>>>> >> As of now, there's no way to do so. There were some efforts on those
>>>> lines but it's been on hold.
>>>> >>
>>>> >> -Anshum
>>>> >>
>>>> >>> On Apr 6, 2016, at 12:21 PM, Don Bosco Durai <bo...@apache.org>
>>>> wrote:
>>>> >>>
>>>> >>> Is there an equivalent of server/scripts/cloud-scripts/zkcli.sh
>>>> -zkhost $zk_host -cmd upconfig -confdir $config_folder -confname
>>>> $config_name using APIs?
>>>> >>>
>>>> >>> I want to bootstrap by uploading the configs via API. Once the
>>>> configs are uploaded, I am now able to do everything else via API.
>>>> >>>
>>>> >>> Thanks
>>>> >>>
>>>> >>> Bosco
>>>> >>>
>>>> >>>
>>>>
>>>>
>>>
>>
Re: Adding configset in SolrCloud via API
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
Therefore, this becomes possible:
http://stackoverflow.com/questions/525212/how-to-run-unix-shell-script-from-java-code
Hackish, but certainly doable... Given there's no API...
On Wed, Apr 6, 2016 at 3:44 PM, John Bickerstaff <jo...@johnbickerstaff.com>
wrote:
> Yup - just tested - that command runs fine with Solr NOT running...
>
> On Wed, Apr 6, 2016 at 3:41 PM, John Bickerstaff <john@johnbickerstaff.com
> > wrote:
>
>> If you can get to the IP addresses from your application, then there's
>> probably a way... Do you mean you're firewalled off or in some other way
>> unable to access the Solr box IP's from your Java application?
>>
>> If you're looking to do "automated build of virtual machines" there are
>> some tools like Vagrant...
>>
>> https://en.wikipedia.org/wiki/Vagrant_(software)
>>
>> Also, you probably don't need to be directly on one of the 'official"
>> SOLR machines to load the configs. I haven't tested, but as long as you
>> have the configs and a VM or server running SOLR, you could do this
>> yourself.
>>
>> The following command (as far as I've been able to tell) ONLY creates the
>> conf "directory" in Zookeeper... nothing else...
>>
>> And, as a matter of fact, I'm almost positive SOLR did not need to be
>> running when I did this, but I did so many variations while trying to
>> figure out how to bring up a new 5.4 collection that I'm not positive - I
>> could be totally wrong on that one.
>>
>> My point is that I uploaded the configs from a machine that had no
>> collection yet... it worked fine to set up then configs in Zookeeper.
>> Later, I issued the create collection command and referenced the config in
>> Zookeeper with the -n(?) flag...
>>
>> sudo /opt/solr/server/scripts/cloud-scripts/zkcli.sh -cmd upconfig
>> -confdir /home/john/conf/ -confname statdx -z 192.168.56.5/solr5_4
>>
>> On Wed, Apr 6, 2016 at 3:26 PM, Don Bosco Durai <bo...@apache.org> wrote:
>>
>>> I have SolrCloud pre-installed. I need to create a collection, but
>>> before that I need to load the config into zookeeper.
>>>
>>> I want to automate the entire process from my Java process which is not
>>> running on any of the servers were SolrCloud is running. In short, I don’t
>>> have access to bin/solr or server/scripts/cloud-scripts, etc from my
>>> application. So I was wondering if there were any way, like uploading a zip
>>> with the configs (schema.xml, solrconfig.xml, etc.). One workaround I can
>>> thinking is of making direct zookeeper calls.
>>>
>>> Anshum, thanks for your reply. I will see if I can find the JIRA.
>>>
>>> Thanks
>>>
>>>
>>> Bosco
>>>
>>>
>>>
>>>
>>> On 4/6/16, 2:17 PM, "Erick Erickson" <er...@gmail.com> wrote:
>>>
>>> >As of Solr 5.5 the bin/solr script can do this, see:
>>> >
>>> https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference
>>> >
>>> >It's still not quite what you're looking for, but uploading arbitrary
>>> >xml scripts through a browser is a security issue, so it's possible
>>> >there will never be an API call to do that.
>>> >
>>> >Best
>>> >Erick
>>> >
>>> >On Wed, Apr 6, 2016 at 1:52 PM, Anshum Gupta <an...@anshumgupta.net>
>>> wrote:
>>> >> As of now, there's no way to do so. There were some efforts on those
>>> lines but it's been on hold.
>>> >>
>>> >> -Anshum
>>> >>
>>> >>> On Apr 6, 2016, at 12:21 PM, Don Bosco Durai <bo...@apache.org>
>>> wrote:
>>> >>>
>>> >>> Is there an equivalent of server/scripts/cloud-scripts/zkcli.sh
>>> -zkhost $zk_host -cmd upconfig -confdir $config_folder -confname
>>> $config_name using APIs?
>>> >>>
>>> >>> I want to bootstrap by uploading the configs via API. Once the
>>> configs are uploaded, I am now able to do everything else via API.
>>> >>>
>>> >>> Thanks
>>> >>>
>>> >>> Bosco
>>> >>>
>>> >>>
>>>
>>>
>>
>
Re: Adding configset in SolrCloud via API
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
Yup - just tested - that command runs fine with Solr NOT running...
On Wed, Apr 6, 2016 at 3:41 PM, John Bickerstaff <jo...@johnbickerstaff.com>
wrote:
> If you can get to the IP addresses from your application, then there's
> probably a way... Do you mean you're firewalled off or in some other way
> unable to access the Solr box IP's from your Java application?
>
> If you're looking to do "automated build of virtual machines" there are
> some tools like Vagrant...
>
> https://en.wikipedia.org/wiki/Vagrant_(software)
>
> Also, you probably don't need to be directly on one of the 'official" SOLR
> machines to load the configs. I haven't tested, but as long as you have
> the configs and a VM or server running SOLR, you could do this yourself.
>
> The following command (as far as I've been able to tell) ONLY creates the
> conf "directory" in Zookeeper... nothing else...
>
> And, as a matter of fact, I'm almost positive SOLR did not need to be
> running when I did this, but I did so many variations while trying to
> figure out how to bring up a new 5.4 collection that I'm not positive - I
> could be totally wrong on that one.
>
> My point is that I uploaded the configs from a machine that had no
> collection yet... it worked fine to set up then configs in Zookeeper.
> Later, I issued the create collection command and referenced the config in
> Zookeeper with the -n(?) flag...
>
> sudo /opt/solr/server/scripts/cloud-scripts/zkcli.sh -cmd upconfig
> -confdir /home/john/conf/ -confname statdx -z 192.168.56.5/solr5_4
>
> On Wed, Apr 6, 2016 at 3:26 PM, Don Bosco Durai <bo...@apache.org> wrote:
>
>> I have SolrCloud pre-installed. I need to create a collection, but before
>> that I need to load the config into zookeeper.
>>
>> I want to automate the entire process from my Java process which is not
>> running on any of the servers were SolrCloud is running. In short, I don’t
>> have access to bin/solr or server/scripts/cloud-scripts, etc from my
>> application. So I was wondering if there were any way, like uploading a zip
>> with the configs (schema.xml, solrconfig.xml, etc.). One workaround I can
>> thinking is of making direct zookeeper calls.
>>
>> Anshum, thanks for your reply. I will see if I can find the JIRA.
>>
>> Thanks
>>
>>
>> Bosco
>>
>>
>>
>>
>> On 4/6/16, 2:17 PM, "Erick Erickson" <er...@gmail.com> wrote:
>>
>> >As of Solr 5.5 the bin/solr script can do this, see:
>> >
>> https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference
>> >
>> >It's still not quite what you're looking for, but uploading arbitrary
>> >xml scripts through a browser is a security issue, so it's possible
>> >there will never be an API call to do that.
>> >
>> >Best
>> >Erick
>> >
>> >On Wed, Apr 6, 2016 at 1:52 PM, Anshum Gupta <an...@anshumgupta.net>
>> wrote:
>> >> As of now, there's no way to do so. There were some efforts on those
>> lines but it's been on hold.
>> >>
>> >> -Anshum
>> >>
>> >>> On Apr 6, 2016, at 12:21 PM, Don Bosco Durai <bo...@apache.org>
>> wrote:
>> >>>
>> >>> Is there an equivalent of server/scripts/cloud-scripts/zkcli.sh
>> -zkhost $zk_host -cmd upconfig -confdir $config_folder -confname
>> $config_name using APIs?
>> >>>
>> >>> I want to bootstrap by uploading the configs via API. Once the
>> configs are uploaded, I am now able to do everything else via API.
>> >>>
>> >>> Thanks
>> >>>
>> >>> Bosco
>> >>>
>> >>>
>>
>>
>
Re: Adding configset in SolrCloud via API
Posted by John Bickerstaff <jo...@johnbickerstaff.com>.
If you can get to the IP addresses from your application, then there's
probably a way... Do you mean you're firewalled off or in some other way
unable to access the Solr box IP's from your Java application?
If you're looking to do "automated build of virtual machines" there are
some tools like Vagrant...
https://en.wikipedia.org/wiki/Vagrant_(software)
Also, you probably don't need to be directly on one of the 'official" SOLR
machines to load the configs. I haven't tested, but as long as you have
the configs and a VM or server running SOLR, you could do this yourself.
The following command (as far as I've been able to tell) ONLY creates the
conf "directory" in Zookeeper... nothing else...
And, as a matter of fact, I'm almost positive SOLR did not need to be
running when I did this, but I did so many variations while trying to
figure out how to bring up a new 5.4 collection that I'm not positive - I
could be totally wrong on that one.
My point is that I uploaded the configs from a machine that had no
collection yet... it worked fine to set up then configs in Zookeeper.
Later, I issued the create collection command and referenced the config in
Zookeeper with the -n(?) flag...
sudo /opt/solr/server/scripts/cloud-scripts/zkcli.sh -cmd upconfig -confdir
/home/john/conf/ -confname statdx -z 192.168.56.5/solr5_4
On Wed, Apr 6, 2016 at 3:26 PM, Don Bosco Durai <bo...@apache.org> wrote:
> I have SolrCloud pre-installed. I need to create a collection, but before
> that I need to load the config into zookeeper.
>
> I want to automate the entire process from my Java process which is not
> running on any of the servers were SolrCloud is running. In short, I don’t
> have access to bin/solr or server/scripts/cloud-scripts, etc from my
> application. So I was wondering if there were any way, like uploading a zip
> with the configs (schema.xml, solrconfig.xml, etc.). One workaround I can
> thinking is of making direct zookeeper calls.
>
> Anshum, thanks for your reply. I will see if I can find the JIRA.
>
> Thanks
>
>
> Bosco
>
>
>
>
> On 4/6/16, 2:17 PM, "Erick Erickson" <er...@gmail.com> wrote:
>
> >As of Solr 5.5 the bin/solr script can do this, see:
> >
> https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference
> >
> >It's still not quite what you're looking for, but uploading arbitrary
> >xml scripts through a browser is a security issue, so it's possible
> >there will never be an API call to do that.
> >
> >Best
> >Erick
> >
> >On Wed, Apr 6, 2016 at 1:52 PM, Anshum Gupta <an...@anshumgupta.net>
> wrote:
> >> As of now, there's no way to do so. There were some efforts on those
> lines but it's been on hold.
> >>
> >> -Anshum
> >>
> >>> On Apr 6, 2016, at 12:21 PM, Don Bosco Durai <bo...@apache.org> wrote:
> >>>
> >>> Is there an equivalent of server/scripts/cloud-scripts/zkcli.sh
> -zkhost $zk_host -cmd upconfig -confdir $config_folder -confname
> $config_name using APIs?
> >>>
> >>> I want to bootstrap by uploading the configs via API. Once the configs
> are uploaded, I am now able to do everything else via API.
> >>>
> >>> Thanks
> >>>
> >>> Bosco
> >>>
> >>>
>
>
Re: Adding configset in SolrCloud via API
Posted by Don Bosco Durai <bo...@apache.org>.
Shawn, thank you. This was exactly what I was looking for.
I am already using SolrJ, so the follow two lines did the job:
ZkConfigManager configManager = new ZkConfigManager(cloudSolrClient.getZkStateReader().getZkClient());
configManager.uploadConfigDir(Paths.get(configPath), configName);
Thanks
Bosco
On 4/6/16, 5:02 PM, "Shawn Heisey" <ap...@elyograg.org> wrote:
>On 4/6/2016 3:26 PM, Don Bosco Durai wrote:
>> I want to automate the entire process from my Java process which is not running on any of the servers were SolrCloud is running. In short, I don’t have access to bin/solr or server/scripts/cloud-scripts, etc from my application. So I was wondering if there were any way, like uploading a zip with the configs (schema.xml, solrconfig.xml, etc.). One workaround I can thinking is of making direct zookeeper calls.
>
>If you're using a dependency management system like maven or ivy, you
>could probably request solr-core, which would let you use ZkCLI or
>ZkConfigManager directly in your own code. That would be a VERY
>significant increase in the size of your app, in the form of dependent
>jars. Although this might work, it's a sledgehammer approach. There
>are 72 direct dependencies for 5.5.0, and some of those have further
>dependencies. Some of solr-core's dependencies are quite large.
>
>http://mvnrepository.com/artifact/org.apache.solr/solr-core
>
>You could look at the code for ZkConfigManager and the classes it uses,
>see how they use zookeeper to send configs, and directly implement the
>zookeeper calls required. Solr implements a wrapper around the
>zookeeper client called SolrZkClient, a wrapping that you might want to
>strip away, so you don't need the solr-core jar and its dependencies.
>This approach requires the most work.
>
>The way the class inheritance is arranged is terrible for user code that
>wants to do SolrCloud config manipulation. I'll see if I can come up
>with something to fix that, but it's not going to happen immediately.
>
>An option that approaches the problem from another direction: Copy
>WEB-INF/lib and other things (like log4j.properties and the logging jars
>in server/lib/ext) to somewhere on your client system and run ZkCLI
>directly as an external process from your own code, just like the zkcli
>script does ... or possibly even using a modified zkcli script. This is
>not as clean as a code-based solution, but it would be relatively easy
>to implement.
>
>Thanks,
>Shawn
>
Re: Adding configset in SolrCloud via API
Posted by Shawn Heisey <ap...@elyograg.org>.
On 4/6/2016 3:26 PM, Don Bosco Durai wrote:
> I want to automate the entire process from my Java process which is not running on any of the servers were SolrCloud is running. In short, I don’t have access to bin/solr or server/scripts/cloud-scripts, etc from my application. So I was wondering if there were any way, like uploading a zip with the configs (schema.xml, solrconfig.xml, etc.). One workaround I can thinking is of making direct zookeeper calls.
If you're using a dependency management system like maven or ivy, you
could probably request solr-core, which would let you use ZkCLI or
ZkConfigManager directly in your own code. That would be a VERY
significant increase in the size of your app, in the form of dependent
jars. Although this might work, it's a sledgehammer approach. There
are 72 direct dependencies for 5.5.0, and some of those have further
dependencies. Some of solr-core's dependencies are quite large.
http://mvnrepository.com/artifact/org.apache.solr/solr-core
You could look at the code for ZkConfigManager and the classes it uses,
see how they use zookeeper to send configs, and directly implement the
zookeeper calls required. Solr implements a wrapper around the
zookeeper client called SolrZkClient, a wrapping that you might want to
strip away, so you don't need the solr-core jar and its dependencies.
This approach requires the most work.
The way the class inheritance is arranged is terrible for user code that
wants to do SolrCloud config manipulation. I'll see if I can come up
with something to fix that, but it's not going to happen immediately.
An option that approaches the problem from another direction: Copy
WEB-INF/lib and other things (like log4j.properties and the logging jars
in server/lib/ext) to somewhere on your client system and run ZkCLI
directly as an external process from your own code, just like the zkcli
script does ... or possibly even using a modified zkcli script. This is
not as clean as a code-based solution, but it would be relatively easy
to implement.
Thanks,
Shawn
Re: Adding configset in SolrCloud via API
Posted by Don Bosco Durai <bo...@apache.org>.
I have SolrCloud pre-installed. I need to create a collection, but before that I need to load the config into zookeeper.
I want to automate the entire process from my Java process which is not running on any of the servers were SolrCloud is running. In short, I don’t have access to bin/solr or server/scripts/cloud-scripts, etc from my application. So I was wondering if there were any way, like uploading a zip with the configs (schema.xml, solrconfig.xml, etc.). One workaround I can thinking is of making direct zookeeper calls.
Anshum, thanks for your reply. I will see if I can find the JIRA.
Thanks
Bosco
On 4/6/16, 2:17 PM, "Erick Erickson" <er...@gmail.com> wrote:
>As of Solr 5.5 the bin/solr script can do this, see:
>https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference
>
>It's still not quite what you're looking for, but uploading arbitrary
>xml scripts through a browser is a security issue, so it's possible
>there will never be an API call to do that.
>
>Best
>Erick
>
>On Wed, Apr 6, 2016 at 1:52 PM, Anshum Gupta <an...@anshumgupta.net> wrote:
>> As of now, there's no way to do so. There were some efforts on those lines but it's been on hold.
>>
>> -Anshum
>>
>>> On Apr 6, 2016, at 12:21 PM, Don Bosco Durai <bo...@apache.org> wrote:
>>>
>>> Is there an equivalent of server/scripts/cloud-scripts/zkcli.sh -zkhost $zk_host -cmd upconfig -confdir $config_folder -confname $config_name using APIs?
>>>
>>> I want to bootstrap by uploading the configs via API. Once the configs are uploaded, I am now able to do everything else via API.
>>>
>>> Thanks
>>>
>>> Bosco
>>>
>>>
Re: Adding configset in SolrCloud via API
Posted by Erick Erickson <er...@gmail.com>.
As of Solr 5.5 the bin/solr script can do this, see:
https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference
It's still not quite what you're looking for, but uploading arbitrary
xml scripts through a browser is a security issue, so it's possible
there will never be an API call to do that.
Best
Erick
On Wed, Apr 6, 2016 at 1:52 PM, Anshum Gupta <an...@anshumgupta.net> wrote:
> As of now, there's no way to do so. There were some efforts on those lines but it's been on hold.
>
> -Anshum
>
>> On Apr 6, 2016, at 12:21 PM, Don Bosco Durai <bo...@apache.org> wrote:
>>
>> Is there an equivalent of server/scripts/cloud-scripts/zkcli.sh -zkhost $zk_host -cmd upconfig -confdir $config_folder -confname $config_name using APIs?
>>
>> I want to bootstrap by uploading the configs via API. Once the configs are uploaded, I am now able to do everything else via API.
>>
>> Thanks
>>
>> Bosco
>>
>>
Re: Adding configset in SolrCloud via API
Posted by Anshum Gupta <an...@anshumgupta.net>.
As of now, there's no way to do so. There were some efforts on those lines but it's been on hold.
-Anshum
> On Apr 6, 2016, at 12:21 PM, Don Bosco Durai <bo...@apache.org> wrote:
>
> Is there an equivalent of server/scripts/cloud-scripts/zkcli.sh -zkhost $zk_host -cmd upconfig -confdir $config_folder -confname $config_name using APIs?
>
> I want to bootstrap by uploading the configs via API. Once the configs are uploaded, I am now able to do everything else via API.
>
> Thanks
>
> Bosco
>
>