You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Puneet Pawaia <pu...@gmail.com> on 2013/11/27 18:15:29 UTC
SolrCloud and 2MB Synonym file
Hi
I am trying to setup a test SolrCloud 4.5.1 implementation. My synonym file
is about 1.6 MB. When I try to add collection to ZooKeeper 3.4.5 on Ubuntu
12.4, it fails because of the 1MB limit of ZooKeeper. Has anyone any
experience with using such synonym files? Can I store them in some other
location other than the Config Folder since the config folder is loaded
into ZooKeeper.
TIA
Puneet Pawaia
Re: SolrCloud and 2MB Synonym file
Posted by Puneet Pawaia <pu...@gmail.com>.
Hi Yago,
Since the value you have mentioned is less than what I have used and it is
working, I won't rock the boat.
Thanks for the env.sh direction. I'll use it next time I need to set things
up.
Regards
Puneet
On Thu, Nov 28, 2013 at 4:01 PM, Yago Riveiro <ya...@gmail.com>wrote:
> In my case I have:
>
> - Zookeeper: a file into conf folder named zookeeper-env.sh with:
>
> #!/usr/bin/env bash
>
> ZOO_ENV="-Djute.maxbuffer=10485761"
>
> The zookeeper-env.sh is loaded automatically by the zkEnv.sh if the file
> exists.
>
> - Solr: the "-Djute.maxbuffer=10485761" param in start command
>
> --
> Yago Riveiro
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Thursday, November 28, 2013 at 10:24 AM, Puneet Pawaia wrote:
>
> > Hi
> >
> > Managed to get it working by adding -Djute.maxbuffer=5242880 to
> > zkServer.sh (http://zkServer.sh), zkCli.sh (http://zkCli.sh) and solr
> startup command.
> >
> > Please let me know if this value for jute.maxbuffer is within acceptable
> > limits
> >
> > Regards
> > Puneet
> >
> >
> > On Thu, Nov 28, 2013 at 6:40 AM, Puneet Pawaia <puneet.pawaia@gmail.com(mailto:
> puneet.pawaia@gmail.com)>wrote:
> >
> > > I am running an ensemble.
> > > Can I get examples of how to use the option? I think there are not many
> > > examples available of the exact usage.
> > >
> > > Regards
> > > Puneet
> > > On 27 Nov 2013 23:23, "Yago Riveiro" <yago.riveiro@gmail.com (mailto:
> yago.riveiro@gmail.com)> wrote:
> > >
> > > > How are you launching Solr?
> > > >
> > > > Do you have an ensemble or you're running zookeeper embedded?
> > > >
> > > > Yes, doc says that jute.maxbuffer is dangerous, but without it you
> can
> > > > stored nothing with more than 1M in zookeeper … and in some point
> you can
> > > > have a clusterstate.json with a size greater than 1M
> > > > --
> > > > Yago Riveiro
> > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > >
> > > >
> > > > On Wednesday, November 27, 2013 at 5:31 PM, Puneet Pawaia wrote:
> > > >
> > > > > Yago, not sure if this is a good idea. Docs say this is dangerous
> stuff.
> > > > >
> > > > > Anyway, not being a linux or java expert, I would appreciate if you
> > > > could
> > > > > point me to an implementation of this.
> > > > >
> > > > > Regards
> > > > > Puneet Pawaia
> > > > > On 27 Nov 2013 22:54, "Yago Riveiro" <yago.riveiro@gmail.com(mailto:
> yago.riveiro@gmail.com) (mailto:
> > > > >
> > > >
> > > > yago.riveiro@gmail.com (mailto:yago.riveiro@gmail.com))> wrote:
> > > > >
> > > > > > You can use the jute.maxbuffer > 1M as a workaround.
> > > > > >
> > > > > > You must set -Djute.maxbuffer in zookeeper and solr to work
> properly
> > > > > >
> > > > > > --
> > > > > > Yago Riveiro
> > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > > >
> > > > > >
> > > > > > On Wednesday, November 27, 2013 at 5:15 PM, Puneet Pawaia wrote:
> > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > I am trying to setup a test SolrCloud 4.5.1 implementation. My
> > > > synonym
> > > > > > file
> > > > > > > is about 1.6 MB. When I try to add collection to ZooKeeper
> 3.4.5 on
> > > > > >
> > > > > >
> > > > > > Ubuntu
> > > > > > > 12.4, it fails because of the 1MB limit of ZooKeeper. Has
> anyone any
> > > > > > > experience with using such synonym files? Can I store them in
> some
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > other
> > > > > > > location other than the Config Folder since the config folder
> is
> > > > > >
> > > > >
> > > >
> > > > loaded
> > > > > > > into ZooKeeper.
> > > > > > >
> > > > > > > TIA
> > > > > > >
> > > > > > > Puneet Pawaia
>
>
Re: SolrCloud and 2MB Synonym file
Posted by Yago Riveiro <ya...@gmail.com>.
In my case I have:
- Zookeeper: a file into conf folder named zookeeper-env.sh with:
#!/usr/bin/env bash
ZOO_ENV="-Djute.maxbuffer=10485761"
The zookeeper-env.sh is loaded automatically by the zkEnv.sh if the file exists.
- Solr: the "-Djute.maxbuffer=10485761" param in start command
--
Yago Riveiro
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Thursday, November 28, 2013 at 10:24 AM, Puneet Pawaia wrote:
> Hi
>
> Managed to get it working by adding -Djute.maxbuffer=5242880 to
> zkServer.sh (http://zkServer.sh), zkCli.sh (http://zkCli.sh) and solr startup command.
>
> Please let me know if this value for jute.maxbuffer is within acceptable
> limits
>
> Regards
> Puneet
>
>
> On Thu, Nov 28, 2013 at 6:40 AM, Puneet Pawaia <puneet.pawaia@gmail.com (mailto:puneet.pawaia@gmail.com)>wrote:
>
> > I am running an ensemble.
> > Can I get examples of how to use the option? I think there are not many
> > examples available of the exact usage.
> >
> > Regards
> > Puneet
> > On 27 Nov 2013 23:23, "Yago Riveiro" <yago.riveiro@gmail.com (mailto:yago.riveiro@gmail.com)> wrote:
> >
> > > How are you launching Solr?
> > >
> > > Do you have an ensemble or you're running zookeeper embedded?
> > >
> > > Yes, doc says that jute.maxbuffer is dangerous, but without it you can
> > > stored nothing with more than 1M in zookeeper … and in some point you can
> > > have a clusterstate.json with a size greater than 1M
> > > --
> > > Yago Riveiro
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > >
> > >
> > > On Wednesday, November 27, 2013 at 5:31 PM, Puneet Pawaia wrote:
> > >
> > > > Yago, not sure if this is a good idea. Docs say this is dangerous stuff.
> > > >
> > > > Anyway, not being a linux or java expert, I would appreciate if you
> > > could
> > > > point me to an implementation of this.
> > > >
> > > > Regards
> > > > Puneet Pawaia
> > > > On 27 Nov 2013 22:54, "Yago Riveiro" <yago.riveiro@gmail.com (mailto:yago.riveiro@gmail.com) (mailto:
> > > >
> > >
> > > yago.riveiro@gmail.com (mailto:yago.riveiro@gmail.com))> wrote:
> > > >
> > > > > You can use the jute.maxbuffer > 1M as a workaround.
> > > > >
> > > > > You must set -Djute.maxbuffer in zookeeper and solr to work properly
> > > > >
> > > > > --
> > > > > Yago Riveiro
> > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > > > >
> > > > >
> > > > > On Wednesday, November 27, 2013 at 5:15 PM, Puneet Pawaia wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > I am trying to setup a test SolrCloud 4.5.1 implementation. My
> > > synonym
> > > > > file
> > > > > > is about 1.6 MB. When I try to add collection to ZooKeeper 3.4.5 on
> > > > >
> > > > >
> > > > > Ubuntu
> > > > > > 12.4, it fails because of the 1MB limit of ZooKeeper. Has anyone any
> > > > > > experience with using such synonym files? Can I store them in some
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> > > other
> > > > > > location other than the Config Folder since the config folder is
> > > > >
> > > >
> > >
> > > loaded
> > > > > > into ZooKeeper.
> > > > > >
> > > > > > TIA
> > > > > >
> > > > > > Puneet Pawaia
Re: SolrCloud and 2MB Synonym file
Posted by Puneet Pawaia <pu...@gmail.com>.
Hi
Managed to get it working by adding -Djute.maxbuffer=5242880 to
zkServer.sh, zkCli.sh and solr startup command.
Please let me know if this value for jute.maxbuffer is within acceptable
limits
Regards
Puneet
On Thu, Nov 28, 2013 at 6:40 AM, Puneet Pawaia <pu...@gmail.com>wrote:
> I am running an ensemble.
> Can I get examples of how to use the option? I think there are not many
> examples available of the exact usage.
>
> Regards
> Puneet
> On 27 Nov 2013 23:23, "Yago Riveiro" <ya...@gmail.com> wrote:
>
>> How are you launching Solr?
>>
>> Do you have an ensemble or you're running zookeeper embedded?
>>
>> Yes, doc says that jute.maxbuffer is dangerous, but without it you can
>> stored nothing with more than 1M in zookeeper … and in some point you can
>> have a clusterstate.json with a size greater than 1M
>> --
>> Yago Riveiro
>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>
>>
>> On Wednesday, November 27, 2013 at 5:31 PM, Puneet Pawaia wrote:
>>
>> > Yago, not sure if this is a good idea. Docs say this is dangerous stuff.
>> >
>> > Anyway, not being a linux or java expert, I would appreciate if you
>> could
>> > point me to an implementation of this.
>> >
>> > Regards
>> > Puneet Pawaia
>> > On 27 Nov 2013 22:54, "Yago Riveiro" <yago.riveiro@gmail.com (mailto:
>> yago.riveiro@gmail.com)> wrote:
>> >
>> > > You can use the jute.maxbuffer > 1M as a workaround.
>> > >
>> > > You must set -Djute.maxbuffer in zookeeper and solr to work properly
>> > >
>> > > --
>> > > Yago Riveiro
>> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>> > >
>> > >
>> > > On Wednesday, November 27, 2013 at 5:15 PM, Puneet Pawaia wrote:
>> > >
>> > > > Hi
>> > > >
>> > > > I am trying to setup a test SolrCloud 4.5.1 implementation. My
>> synonym
>> > > file
>> > > > is about 1.6 MB. When I try to add collection to ZooKeeper 3.4.5 on
>> > >
>> > > Ubuntu
>> > > > 12.4, it fails because of the 1MB limit of ZooKeeper. Has anyone any
>> > > > experience with using such synonym files? Can I store them in some
>> other
>> > > > location other than the Config Folder since the config folder is
>> loaded
>> > > > into ZooKeeper.
>> > > >
>> > > > TIA
>> > > >
>> > > > Puneet Pawaia
>>
>>
Re: SolrCloud and 2MB Synonym file
Posted by Puneet Pawaia <pu...@gmail.com>.
I am running an ensemble.
Can I get examples of how to use the option? I think there are not many
examples available of the exact usage.
Regards
Puneet
On 27 Nov 2013 23:23, "Yago Riveiro" <ya...@gmail.com> wrote:
> How are you launching Solr?
>
> Do you have an ensemble or you're running zookeeper embedded?
>
> Yes, doc says that jute.maxbuffer is dangerous, but without it you can
> stored nothing with more than 1M in zookeeper … and in some point you can
> have a clusterstate.json with a size greater than 1M
> --
> Yago Riveiro
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Wednesday, November 27, 2013 at 5:31 PM, Puneet Pawaia wrote:
>
> > Yago, not sure if this is a good idea. Docs say this is dangerous stuff.
> >
> > Anyway, not being a linux or java expert, I would appreciate if you could
> > point me to an implementation of this.
> >
> > Regards
> > Puneet Pawaia
> > On 27 Nov 2013 22:54, "Yago Riveiro" <yago.riveiro@gmail.com (mailto:
> yago.riveiro@gmail.com)> wrote:
> >
> > > You can use the jute.maxbuffer > 1M as a workaround.
> > >
> > > You must set -Djute.maxbuffer in zookeeper and solr to work properly
> > >
> > > --
> > > Yago Riveiro
> > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> > >
> > >
> > > On Wednesday, November 27, 2013 at 5:15 PM, Puneet Pawaia wrote:
> > >
> > > > Hi
> > > >
> > > > I am trying to setup a test SolrCloud 4.5.1 implementation. My
> synonym
> > > file
> > > > is about 1.6 MB. When I try to add collection to ZooKeeper 3.4.5 on
> > >
> > > Ubuntu
> > > > 12.4, it fails because of the 1MB limit of ZooKeeper. Has anyone any
> > > > experience with using such synonym files? Can I store them in some
> other
> > > > location other than the Config Folder since the config folder is
> loaded
> > > > into ZooKeeper.
> > > >
> > > > TIA
> > > >
> > > > Puneet Pawaia
>
>
Re: SolrCloud and 2MB Synonym file
Posted by Yago Riveiro <ya...@gmail.com>.
How are you launching Solr?
Do you have an ensemble or you're running zookeeper embedded?
Yes, doc says that jute.maxbuffer is dangerous, but without it you can stored nothing with more than 1M in zookeeper … and in some point you can have a clusterstate.json with a size greater than 1M
--
Yago Riveiro
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Wednesday, November 27, 2013 at 5:31 PM, Puneet Pawaia wrote:
> Yago, not sure if this is a good idea. Docs say this is dangerous stuff.
>
> Anyway, not being a linux or java expert, I would appreciate if you could
> point me to an implementation of this.
>
> Regards
> Puneet Pawaia
> On 27 Nov 2013 22:54, "Yago Riveiro" <yago.riveiro@gmail.com (mailto:yago.riveiro@gmail.com)> wrote:
>
> > You can use the jute.maxbuffer > 1M as a workaround.
> >
> > You must set -Djute.maxbuffer in zookeeper and solr to work properly
> >
> > --
> > Yago Riveiro
> > Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> >
> >
> > On Wednesday, November 27, 2013 at 5:15 PM, Puneet Pawaia wrote:
> >
> > > Hi
> > >
> > > I am trying to setup a test SolrCloud 4.5.1 implementation. My synonym
> > file
> > > is about 1.6 MB. When I try to add collection to ZooKeeper 3.4.5 on
> >
> > Ubuntu
> > > 12.4, it fails because of the 1MB limit of ZooKeeper. Has anyone any
> > > experience with using such synonym files? Can I store them in some other
> > > location other than the Config Folder since the config folder is loaded
> > > into ZooKeeper.
> > >
> > > TIA
> > >
> > > Puneet Pawaia
Re: SolrCloud and 2MB Synonym file
Posted by Timothy Potter <th...@gmail.com>.
I'm curious how much compression you get with your synonym file using
something basic like gzip? If significant, would it make sense to
store the compressed syn file in ZooKeeper (or any other metadata you
need to distribute around the cluster)? This would require the code
that reads the syn file from ZooKeeper to be able to de-compress it.
Seems like this would be a nice-to-have in SolrCloud in general - the
ability to read / write files to ZooKeeper in compressed format.
Tim
On Wed, Nov 27, 2013 at 11:29 AM, Mark Miller <ma...@gmail.com> wrote:
> They are just trying to keep users from using ZK in a bad way. Storing and accessing a ton of huge files is not what ZooKeeper was designed for. A 1MB limit is a fairly arbitrary limiter to make sure you don’t shoot yourself in the foot and store lots of large files. With modern networks and hardware, setting it to 3MB and uploading your 2MB syn file is not going to be a problem. Solr doesn’t read and write those files often, nor use ZooKeeper much at all in a stable state. Upping that limit and putting in a few config files that are a few MB is not going to break anything.
>
> - Mark
>
> On Nov 27, 2013, at 12:31 PM, Puneet Pawaia <pu...@gmail.com> wrote:
>
>> Yago, not sure if this is a good idea. Docs say this is dangerous stuff.
>>
>> Anyway, not being a linux or java expert, I would appreciate if you could
>> point me to an implementation of this.
>>
>> Regards
>> Puneet Pawaia
>> On 27 Nov 2013 22:54, "Yago Riveiro" <ya...@gmail.com> wrote:
>>
>>> You can use the jute.maxbuffer > 1M as a workaround.
>>>
>>> You must set -Djute.maxbuffer in zookeeper and solr to work properly
>>>
>>> --
>>> Yago Riveiro
>>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>>
>>>
>>> On Wednesday, November 27, 2013 at 5:15 PM, Puneet Pawaia wrote:
>>>
>>>> Hi
>>>>
>>>> I am trying to setup a test SolrCloud 4.5.1 implementation. My synonym
>>> file
>>>> is about 1.6 MB. When I try to add collection to ZooKeeper 3.4.5 on
>>> Ubuntu
>>>> 12.4, it fails because of the 1MB limit of ZooKeeper. Has anyone any
>>>> experience with using such synonym files? Can I store them in some other
>>>> location other than the Config Folder since the config folder is loaded
>>>> into ZooKeeper.
>>>>
>>>> TIA
>>>>
>>>> Puneet Pawaia
>>>
>>>
>
Re: SolrCloud and 2MB Synonym file
Posted by Mark Miller <ma...@gmail.com>.
They are just trying to keep users from using ZK in a bad way. Storing and accessing a ton of huge files is not what ZooKeeper was designed for. A 1MB limit is a fairly arbitrary limiter to make sure you don’t shoot yourself in the foot and store lots of large files. With modern networks and hardware, setting it to 3MB and uploading your 2MB syn file is not going to be a problem. Solr doesn’t read and write those files often, nor use ZooKeeper much at all in a stable state. Upping that limit and putting in a few config files that are a few MB is not going to break anything.
- Mark
On Nov 27, 2013, at 12:31 PM, Puneet Pawaia <pu...@gmail.com> wrote:
> Yago, not sure if this is a good idea. Docs say this is dangerous stuff.
>
> Anyway, not being a linux or java expert, I would appreciate if you could
> point me to an implementation of this.
>
> Regards
> Puneet Pawaia
> On 27 Nov 2013 22:54, "Yago Riveiro" <ya...@gmail.com> wrote:
>
>> You can use the jute.maxbuffer > 1M as a workaround.
>>
>> You must set -Djute.maxbuffer in zookeeper and solr to work properly
>>
>> --
>> Yago Riveiro
>> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>>
>>
>> On Wednesday, November 27, 2013 at 5:15 PM, Puneet Pawaia wrote:
>>
>>> Hi
>>>
>>> I am trying to setup a test SolrCloud 4.5.1 implementation. My synonym
>> file
>>> is about 1.6 MB. When I try to add collection to ZooKeeper 3.4.5 on
>> Ubuntu
>>> 12.4, it fails because of the 1MB limit of ZooKeeper. Has anyone any
>>> experience with using such synonym files? Can I store them in some other
>>> location other than the Config Folder since the config folder is loaded
>>> into ZooKeeper.
>>>
>>> TIA
>>>
>>> Puneet Pawaia
>>
>>
Re: SolrCloud and 2MB Synonym file
Posted by Puneet Pawaia <pu...@gmail.com>.
Yago, not sure if this is a good idea. Docs say this is dangerous stuff.
Anyway, not being a linux or java expert, I would appreciate if you could
point me to an implementation of this.
Regards
Puneet Pawaia
On 27 Nov 2013 22:54, "Yago Riveiro" <ya...@gmail.com> wrote:
> You can use the jute.maxbuffer > 1M as a workaround.
>
> You must set -Djute.maxbuffer in zookeeper and solr to work properly
>
> --
> Yago Riveiro
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Wednesday, November 27, 2013 at 5:15 PM, Puneet Pawaia wrote:
>
> > Hi
> >
> > I am trying to setup a test SolrCloud 4.5.1 implementation. My synonym
> file
> > is about 1.6 MB. When I try to add collection to ZooKeeper 3.4.5 on
> Ubuntu
> > 12.4, it fails because of the 1MB limit of ZooKeeper. Has anyone any
> > experience with using such synonym files? Can I store them in some other
> > location other than the Config Folder since the config folder is loaded
> > into ZooKeeper.
> >
> > TIA
> >
> > Puneet Pawaia
>
>
Re: SolrCloud and 2MB Synonym file
Posted by Yago Riveiro <ya...@gmail.com>.
You can use the jute.maxbuffer > 1M as a workaround.
You must set -Djute.maxbuffer in zookeeper and solr to work properly
--
Yago Riveiro
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Wednesday, November 27, 2013 at 5:15 PM, Puneet Pawaia wrote:
> Hi
>
> I am trying to setup a test SolrCloud 4.5.1 implementation. My synonym file
> is about 1.6 MB. When I try to add collection to ZooKeeper 3.4.5 on Ubuntu
> 12.4, it fails because of the 1MB limit of ZooKeeper. Has anyone any
> experience with using such synonym files? Can I store them in some other
> location other than the Config Folder since the config folder is loaded
> into ZooKeeper.
>
> TIA
>
> Puneet Pawaia