You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Eric Pugh <ep...@opensourceconnections.com> on 2012/08/14 22:11:12 UTC
Custom Jars for a config in the Solr Cloud world..
I've got a Solr instance with a number of cores that are each configured by upload the configuration information to ZooKeeper. The newest index needs the UIMA jars. Normally I would put them in the core's /lib directory, but since I am only accessing my server via ZooKeeper, I don't have that directory as an option.
I know I could manually upload the jars onto the server, and then put some sort of path to them, but I'm hoping to manage all uploading of core specific configurations (and jars) via ZooKeeper. I'm wondering if I am missing something in this new ZooKeeper enabled world....? Just for fun, I'm going to try and put the ~ 2 MB worth of Jars inside my /conf/ directory and then upload through ZooKeeper to see what happens.
Eric
-----------------------------------------------------
Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com
Co-Author: Apache Solr 3 Enterprise Search Server available from http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
Re: Custom Jars for a config in the Solr Cloud world..
Posted by Chris Hostetter <ho...@fucit.org>.
: by upload the configuration information to ZooKeeper. The newest index
: needs the UIMA jars. Normally I would put them in the core's /lib
: directory, but since I am only accessing my server via ZooKeeper, I
: don't have that directory as an option.
:
: I know I could manually upload the jars onto the server, and then put
: some sort of path to them, but I'm hoping to manage all uploading of
: core specific configurations (and jars) via ZooKeeper. I'm wondering if
i would use one or more explicit <lib ... /> directives in your
solrconfig.xml file, pointing at absolute paths where you consistently
upload these jars on every machine (or better still: use sharedLib in
solr.xml) rather then the implicit ./lib per core.
that way no matter how many collections you create on these machines (or
how many solr instances), the configs can all point to a single uniform
place on the machine to find these jars instead of needing ot copy them
arround.
yes: you still have to upload them to the machines yourself, but you also
have to upload solr to these machines yourself -- code needs deployed,
even if it's a plugin.
>> ... maybe I just need to tweak my <lib/> definitions in my
>> solrconfig.xml to look for all the places that Jars may exist, even
>> though on my local box it's different then on my integration Solr box.
doesn't sound very cloud-ish to have diff stuff installed in diff places
on diff machines ... but you could use a system property to specify the
basepath for all your libs and then override it on your dev machine.
-Hoss
Re: Custom Jars for a config in the Solr Cloud world..
Posted by Eric Pugh <ep...@opensourceconnections.com>.
And I can now confirm that yes, ZooKeeper blows up when I attempted to add all the UIMA and content extraction jars to my conf/ directory in ZooKeeper! A couple small jars did upload, and then it started sending back "java.io.IOException: Broken pipe" errors.
So any thoughts on the best way to manage Jars that seem like they should be part of your config? Small jars I think will work, and maybe I just need to tweak my <lib/> definitions in my solrconfig.xml to look for all the places that Jars may exist, even though on my local box it's different then on my integration Solr box. Just seems a bit messy ;-)
Eric
On Aug 14, 2012, at 4:40 PM, Jack Krupansky wrote:
> Dear Eric The Brave,
>
> As per the wiki:"znodes are limited to the amount of data that they can have. ZooKeeper was designed to store coordination data: status information, configuration, location information, etc. This kind of meta-information is usually measured in kilobytes, if not bytes. ZooKeeper has a built-in sanity check of 1M, to prevent it from being used as a large data store, but in general it is used to store much smaller pieces of data."
>
> See:
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ProjectDescription
>
> Also:"jute.maxbuffer: (Java system property: jute.maxbuffer)
> This option can only be set as a Java system property. There is no zookeeper prefix on it. It specifies the maximum size of the data that can be stored in a znode. The default is 0xfffff, or just under 1M. If this option is changed, the system property must be set on all servers and clients otherwise problems will arise. This is really a sanity check. ZooKeeper is designed to store data on the order of kilobytes in size."
>
> See:
> http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html
>
> -- Jack Krupansky
>
> -----Original Message----- From: Eric Pugh
> Sent: Tuesday, August 14, 2012 4:11 PM
> To: solr-user@lucene.apache.org
> Subject: Custom Jars for a config in the Solr Cloud world..
>
> I've got a Solr instance with a number of cores that are each configured by upload the configuration information to ZooKeeper. The newest index needs the UIMA jars. Normally I would put them in the core's /lib directory, but since I am only accessing my server via ZooKeeper, I don't have that directory as an option.
>
> I know I could manually upload the jars onto the server, and then put some sort of path to them, but I'm hoping to manage all uploading of core specific configurations (and jars) via ZooKeeper. I'm wondering if I am missing something in this new ZooKeeper enabled world....? Just for fun, I'm going to try and put the ~ 2 MB worth of Jars inside my /conf/ directory and then upload through ZooKeeper to see what happens.
>
> Eric
>
>
> -----------------------------------------------------
> Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com
> Co-Author: Apache Solr 3 Enterprise Search Server available from http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
> This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
>
>
>
>
>
>
>
>
>
>
-----------------------------------------------------
Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com
Co-Author: Apache Solr 3 Enterprise Search Server available from http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
This e-mail and all contents, including attachments, is considered to be Company Confidential unless explicitly stated otherwise, regardless of whether attachments are marked as such.
Re: Custom Jars for a config in the Solr Cloud world..
Posted by Jack Krupansky <ja...@basetechnology.com>.
Dear Eric The Brave,
As per the wiki:"znodes are limited to the amount of data that they can
have. ZooKeeper was designed to store coordination data: status information,
configuration, location information, etc. This kind of meta-information is
usually measured in kilobytes, if not bytes. ZooKeeper has a built-in sanity
check of 1M, to prevent it from being used as a large data store, but in
general it is used to store much smaller pieces of data."
See:
https://cwiki.apache.org/confluence/display/ZOOKEEPER/ProjectDescription
Also:"jute.maxbuffer: (Java system property: jute.maxbuffer)
This option can only be set as a Java system property. There is no zookeeper
prefix on it. It specifies the maximum size of the data that can be stored
in a znode. The default is 0xfffff, or just under 1M. If this option is
changed, the system property must be set on all servers and clients
otherwise problems will arise. This is really a sanity check. ZooKeeper is
designed to store data on the order of kilobytes in size."
See:
http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html
-- Jack Krupansky
-----Original Message-----
From: Eric Pugh
Sent: Tuesday, August 14, 2012 4:11 PM
To: solr-user@lucene.apache.org
Subject: Custom Jars for a config in the Solr Cloud world..
I've got a Solr instance with a number of cores that are each configured by
upload the configuration information to ZooKeeper. The newest index needs
the UIMA jars. Normally I would put them in the core's /lib directory, but
since I am only accessing my server via ZooKeeper, I don't have that
directory as an option.
I know I could manually upload the jars onto the server, and then put some
sort of path to them, but I'm hoping to manage all uploading of core
specific configurations (and jars) via ZooKeeper. I'm wondering if I am
missing something in this new ZooKeeper enabled world....? Just for fun,
I'm going to try and put the ~ 2 MB worth of Jars inside my /conf/ directory
and then upload through ZooKeeper to see what happens.
Eric
-----------------------------------------------------
Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 |
http://www.opensourceconnections.com
Co-Author: Apache Solr 3 Enterprise Search Server available from
http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
This e-mail and all contents, including attachments, is considered to be
Company Confidential unless explicitly stated otherwise, regardless of
whether attachments are marked as such.