You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Markus Jelsma <ma...@openindex.io> on 2013/08/02 14:18:59 UTC
Large config files in SolrCloud
Hi,
I have a few very large configuration files but it doens't work in cloud mode due to the KeeperException$ConnectionLossException. All 10 Solr nodes run trunk and have jute.maxbuffer set to 5242880 (5MB). I can confirm it is set properly by looking at the args in the Solr GUI. All Zookeepers have exactly the same key/value set, i can confirm this by looking with ps at the process, it is really there, the first parameter.
But it doesn't work!
I have had it working once but it doesn't like me anymore. Putting the config files in the node's sharedLib or the core's lib directory doesn't work either, the files are clearly loaded according to the logs but the TokenFilters cannot access them and complain about a config file not being found.
I'm out of ideas, any to share?
Thanks,
Markus
RE: Large config files in SolrCloud
Posted by Markus Jelsma <ma...@openindex.io>.
With SOLR-5115 there's support for forcing ZkResourceLoader to use SolrResourceLoader using file:/// prefix in your schema. This forces Solr to load files from the FS as usual.
https://issues.apache.org/jira/browse/SOLR-5115
-----Original message-----
> From:Markus Jelsma <ma...@openindex.io>
> Sent: Friday 2nd August 2013 15:27
> To: solr-user@lucene.apache.org
> Subject: RE: Large config files in SolrCloud
>
> Ok, i managed to load a config file from the node's sharedLib directory by pointing it to `../../lib/file`. This works fine in normal mode, in cloud mode all it does is attempting to find it in Zookeeper by trying /configs/COLLECTION_NAME/path. Properties such as ${solr.home} are not recognized in the schema, it seems.
>
> Any idea on how to force Solr in cloud mode to grab the file from the local FS?
>
> Thanks,
> Markus
>
> -----Original message-----
> > From:Markus Jelsma <ma...@openindex.io>
> > Sent: Friday 2nd August 2013 14:34
> > To: solr-user@lucene.apache.org
> > Subject: RE: Large config files in SolrCloud
> >
> > Yes, all the usual config files are well under 1MB and work as expected. This file is under 2MB and the limit i set is 5MB. Setting jute.maxbuffer (all lowercase) did work during a test a long time ago but we'd like to put the new features in production and we're stuck at this trivial issue :)
> >
> > Thanks
> >
> > -----Original message-----
> > > From:Erick Erickson <er...@gmail.com>
> > > Sent: Friday 2nd August 2013 14:28
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: Large config files in SolrCloud
> > >
> > > Hmmm, does it work with smaller config files? There's been a limit of 1M
> > > for ZK files, and I'm wondering if your setup would work with, say, 2M
> > > configs as a check that it's "something else" rather than just the 1M limit.
> > >
> > > FWIW,
> > > Erick
> > >
> > >
> > > On Fri, Aug 2, 2013 at 8:18 AM, Markus Jelsma <ma...@openindex.io>wrote:
> > >
> > > > Hi,
> > > >
> > > > I have a few very large configuration files but it doens't work in cloud
> > > > mode due to the KeeperException$ConnectionLossException. All 10 Solr nodes
> > > > run trunk and have jute.maxbuffer set to 5242880 (5MB). I can confirm it is
> > > > set properly by looking at the args in the Solr GUI. All Zookeepers have
> > > > exactly the same key/value set, i can confirm this by looking with ps at
> > > > the process, it is really there, the first parameter.
> > > >
> > > > But it doesn't work!
> > > >
> > > > I have had it working once but it doesn't like me anymore. Putting the
> > > > config files in the node's sharedLib or the core's lib directory doesn't
> > > > work either, the files are clearly loaded according to the logs but the
> > > > TokenFilters cannot access them and complain about a config file not being
> > > > found.
> > > >
> > > > I'm out of ideas, any to share?
> > > >
> > > > Thanks,
> > > > Markus
> > > >
> > >
> >
>
RE: Large config files in SolrCloud
Posted by Markus Jelsma <ma...@openindex.io>.
Ok, i managed to load a config file from the node's sharedLib directory by pointing it to `../../lib/file`. This works fine in normal mode, in cloud mode all it does is attempting to find it in Zookeeper by trying /configs/COLLECTION_NAME/path. Properties such as ${solr.home} are not recognized in the schema, it seems.
Any idea on how to force Solr in cloud mode to grab the file from the local FS?
Thanks,
Markus
-----Original message-----
> From:Markus Jelsma <ma...@openindex.io>
> Sent: Friday 2nd August 2013 14:34
> To: solr-user@lucene.apache.org
> Subject: RE: Large config files in SolrCloud
>
> Yes, all the usual config files are well under 1MB and work as expected. This file is under 2MB and the limit i set is 5MB. Setting jute.maxbuffer (all lowercase) did work during a test a long time ago but we'd like to put the new features in production and we're stuck at this trivial issue :)
>
> Thanks
>
> -----Original message-----
> > From:Erick Erickson <er...@gmail.com>
> > Sent: Friday 2nd August 2013 14:28
> > To: solr-user@lucene.apache.org
> > Subject: Re: Large config files in SolrCloud
> >
> > Hmmm, does it work with smaller config files? There's been a limit of 1M
> > for ZK files, and I'm wondering if your setup would work with, say, 2M
> > configs as a check that it's "something else" rather than just the 1M limit.
> >
> > FWIW,
> > Erick
> >
> >
> > On Fri, Aug 2, 2013 at 8:18 AM, Markus Jelsma <ma...@openindex.io>wrote:
> >
> > > Hi,
> > >
> > > I have a few very large configuration files but it doens't work in cloud
> > > mode due to the KeeperException$ConnectionLossException. All 10 Solr nodes
> > > run trunk and have jute.maxbuffer set to 5242880 (5MB). I can confirm it is
> > > set properly by looking at the args in the Solr GUI. All Zookeepers have
> > > exactly the same key/value set, i can confirm this by looking with ps at
> > > the process, it is really there, the first parameter.
> > >
> > > But it doesn't work!
> > >
> > > I have had it working once but it doesn't like me anymore. Putting the
> > > config files in the node's sharedLib or the core's lib directory doesn't
> > > work either, the files are clearly loaded according to the logs but the
> > > TokenFilters cannot access them and complain about a config file not being
> > > found.
> > >
> > > I'm out of ideas, any to share?
> > >
> > > Thanks,
> > > Markus
> > >
> >
>
Re: Large config files in SolrCloud
Posted by Erick Erickson <er...@gmail.com>.
Hmmm, does it work with smaller config files? There's been a limit of 1M
for ZK files, and I'm wondering if your setup would work with, say, 2M
configs as a check that it's "something else" rather than just the 1M limit.
FWIW,
Erick
On Fri, Aug 2, 2013 at 8:18 AM, Markus Jelsma <ma...@openindex.io>wrote:
> Hi,
>
> I have a few very large configuration files but it doens't work in cloud
> mode due to the KeeperException$ConnectionLossException. All 10 Solr nodes
> run trunk and have jute.maxbuffer set to 5242880 (5MB). I can confirm it is
> set properly by looking at the args in the Solr GUI. All Zookeepers have
> exactly the same key/value set, i can confirm this by looking with ps at
> the process, it is really there, the first parameter.
>
> But it doesn't work!
>
> I have had it working once but it doesn't like me anymore. Putting the
> config files in the node's sharedLib or the core's lib directory doesn't
> work either, the files are clearly loaded according to the logs but the
> TokenFilters cannot access them and complain about a config file not being
> found.
>
> I'm out of ideas, any to share?
>
> Thanks,
> Markus
>
RE: Large config files in SolrCloud
Posted by Markus Jelsma <ma...@openindex.io>.
Yes, all the usual config files are well under 1MB and work as expected. This file is under 2MB and the limit i set is 5MB. Setting jute.maxbuffer (all lowercase) did work during a test a long time ago but we'd like to put the new features in production and we're stuck at this trivial issue :)
Thanks
-----Original message-----
> From:Erick Erickson <er...@gmail.com>
> Sent: Friday 2nd August 2013 14:28
> To: solr-user@lucene.apache.org
> Subject: Re: Large config files in SolrCloud
>
> Hmmm, does it work with smaller config files? There's been a limit of 1M
> for ZK files, and I'm wondering if your setup would work with, say, 2M
> configs as a check that it's "something else" rather than just the 1M limit.
>
> FWIW,
> Erick
>
>
> On Fri, Aug 2, 2013 at 8:18 AM, Markus Jelsma <ma...@openindex.io>wrote:
>
> > Hi,
> >
> > I have a few very large configuration files but it doens't work in cloud
> > mode due to the KeeperException$ConnectionLossException. All 10 Solr nodes
> > run trunk and have jute.maxbuffer set to 5242880 (5MB). I can confirm it is
> > set properly by looking at the args in the Solr GUI. All Zookeepers have
> > exactly the same key/value set, i can confirm this by looking with ps at
> > the process, it is really there, the first parameter.
> >
> > But it doesn't work!
> >
> > I have had it working once but it doesn't like me anymore. Putting the
> > config files in the node's sharedLib or the core's lib directory doesn't
> > work either, the files are clearly loaded according to the logs but the
> > TokenFilters cannot access them and complain about a config file not being
> > found.
> >
> > I'm out of ideas, any to share?
> >
> > Thanks,
> > Markus
> >
>