You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Fadi Mohsen <fa...@gmail.com> on 2013/01/24 16:02:23 UTC
Submit schema definition using curl via SOLR
Hi, We would like to use Solr to index statistics from any Java module in
our production environment.
Applications have to can create collections and index data on demand, so my
initial thought is to use different HTTP methods to accomplish a collection
in cluster and then right away start HTTP POST documents, but the issue
here is the schema.xml.
Is it possible to HTTP POST the schema via Solr to Zookeeper?
Or do I have to know about other service host/IP than SOLR, such as
ZooKeeper (wanted to understand whether there is a way to avoid knowing
about zookeeper in production.)?
This must be a duplicate of another question, excuse me in advance.
Regards
Fadi
Re: Submit schema definition using curl via SOLR
Posted by Per Steffensen <st...@designware.dk>.
On 1/24/13 11:22 PM, Fadi Mohsen wrote:
> Thanks Per, would the first approach involve restarting Solr?
>
>
Of course ZK need to run in order to load the config into ZK. Solr nodes
do not need to run. If they do I couldnt imagine that they need to be
restarted in order to take advantage of new config for future collections.
Re: Submit schema definition using curl via SOLR
Posted by Mark Miller <ma...@gmail.com>.
On Jan 24, 2013, at 5:22 PM, Fadi Mohsen <fa...@gmail.com> wrote:
>
> The reason we would like to avoid Zookeeper are
> * due to lack of knowledge.
> * the amount of work/scripting for developers per module and release
> documentation.
> * the extra steps of patching ZK nodes for QA and operations.
>
> ZkCLI is a nice tool, but then instead of interacting with one service over
> HTTP, the application needs:
> * extra jar files
We should address this I think - it really shouldn't require anymore than the SolrJ jars. Currently it also requires the core jars. Still not as minimal as just curl posting I know.
Testing and reporting on the issue I posted, as well as discussion around expanding it, will likely help pushing those features forward.
- Mark
Re: Submit schema definition using curl via SOLR
Posted by Fadi Mohsen <fa...@gmail.com>.
Thanks Per, would the first approach involve restarting Solr?
Thanks Mark, that's great, Ill try checkout and apply patches from ticket
to understand further.
The reason we would like to avoid Zookeeper are
* due to lack of knowledge.
* the amount of work/scripting for developers per module and release
documentation.
* the extra steps of patching ZK nodes for QA and operations.
ZkCLI is a nice tool, but then instead of interacting with one service over
HTTP, the application needs:
* extra jar files
* know ZK hostname/IP and port (different in each
dev/qa/systest/accept/production environment), which is per module a one to
much configuration step.
On Thu, Jan 24, 2013 at 7:18 PM, Mark Miller <ma...@gmail.com> wrote:
>
> On Jan 24, 2013, at 10:02 AM, Fadi Mohsen <fa...@gmail.com> wrote:
>
> > Hi, We would like to use Solr to index statistics from any Java module in
> > our production environment.
> >
> > Applications have to can create collections and index data on demand, so
> my
> > initial thought is to use different HTTP methods to accomplish a
> collection
> > in cluster and then right away start HTTP POST documents, but the issue
> > here is the schema.xml.
> > Is it possible to HTTP POST the schema via Solr to Zookeeper?
>
> I've done some work towards this at
> https://issues.apache.org/jira/browse/SOLR-4193
>
> >
> > Or do I have to know about other service host/IP than SOLR, such as
> > ZooKeeper (wanted to understand whether there is a way to avoid knowing
> > about zookeeper in production.)?
>
> I wouldn't try to avoid it - it's probably simpler to deal with than you
> think.
>
> It's also pretty easy to use
> http://wiki.apache.org/solr/SolrCloud#Command_Line_Util to upload a new
> schema.xml - then just Collections API reload command. Two lines in a
> script.
>
> - Mark
>
>
Re: Submit schema definition using curl via SOLR
Posted by Mark Miller <ma...@gmail.com>.
On Jan 24, 2013, at 10:02 AM, Fadi Mohsen <fa...@gmail.com> wrote:
> Hi, We would like to use Solr to index statistics from any Java module in
> our production environment.
>
> Applications have to can create collections and index data on demand, so my
> initial thought is to use different HTTP methods to accomplish a collection
> in cluster and then right away start HTTP POST documents, but the issue
> here is the schema.xml.
> Is it possible to HTTP POST the schema via Solr to Zookeeper?
I've done some work towards this at https://issues.apache.org/jira/browse/SOLR-4193
>
> Or do I have to know about other service host/IP than SOLR, such as
> ZooKeeper (wanted to understand whether there is a way to avoid knowing
> about zookeeper in production.)?
I wouldn't try to avoid it - it's probably simpler to deal with than you think.
It's also pretty easy to use http://wiki.apache.org/solr/SolrCloud#Command_Line_Util to upload a new schema.xml - then just Collections API reload command. Two lines in a script.
- Mark
Re: Submit schema definition using curl via SOLR
Posted by Per Steffensen <st...@designware.dk>.
On 1/24/13 4:51 PM, Per Steffensen wrote:
>
> 2) or You can have an Solr node (server) load a "Solr config" into ZK
> during startup by adding collection.configName and bootstrap_confdir
> VM params - something like this
> java -DzkHost=<zk_connection_str>
> -Dcollection.configName=<logical_solr_config_name>
> -Dbootstrap_confdir=<path_to_solr_config_dir> -jar start.jar
Well <logical_solr_config_name> instead of edr_sms_conf, of course
Re: Submit schema definition using curl via SOLR
Posted by Per Steffensen <st...@designware.dk>.
Basically uploading a "Solr config" (including schema.xml,
solrconfig.xml etc.) is an operation different from creating
collections. When creating a collection (e.g. using the Collection API)
you reference the (already existing) "Solr config" it needs to use.
Collections can share "Solr config"s. I know of at least two ways to
load a "Solr config" into ZK using Solr-tools.
1) You can use ZkCLI tool (of course ZK needs to be started) - something
like this
mkdir -p "${SOLR_INSTALL}/example/webapps/temp"
cp "${SOLR_INSTALL}/example/webapps/solr.war"
"${SOLR_INSTALL}/example/webapps/temp"
cd "${SOLR_INSTALL}/example/webapps/temp"
jar -xf "solr.war"
java -classpath "${SOLR_INSTALL}/example/webapps/temp/WEB-INF/lib/*"
org.apache.solr.cloud.ZkCLI -cmd upconfig -confdir
<path_to_solr_config_dir> -confname <logical_solr_config_name> --zkhost
<zk_connection_str>
rm -rf "${SOLR_INSTALL}/example/webapps/temp"
Believe there is also a zkcli.sh tool
2) or You can have an Solr node (server) load a "Solr config" into ZK
during startup by adding collection.configName and bootstrap_confdir VM
params - something like this
java -DzkHost=<zk_connection_str> -Dcollection.configName=edr_sms_conf
-Dbootstrap_confdir=<path_to_solr_config_dir> -jar start.jar
I prefer 1) for several reasons.
Regards, Per Steffensen
On 1/24/13 4:02 PM, Fadi Mohsen wrote:
> Hi, We would like to use Solr to index statistics from any Java module in
> our production environment.
>
> Applications have to can create collections and index data on demand, so my
> initial thought is to use different HTTP methods to accomplish a collection
> in cluster and then right away start HTTP POST documents, but the issue
> here is the schema.xml.
> Is it possible to HTTP POST the schema via Solr to Zookeeper?
>
> Or do I have to know about other service host/IP than SOLR, such as
> ZooKeeper (wanted to understand whether there is a way to avoid knowing
> about zookeeper in production.)?
>
> This must be a duplicate of another question, excuse me in advance.
>
> Regards
> Fadi
>