You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by yriveiro <ya...@gmail.com> on 2013/03/30 18:49:45 UTC
clusterstate.json size
Hi,
Is there a size limitation for the clusterstate file?
I can't create more collections for my cluster I have no error but the
CREATE command not return any response.
I read in the past that the max size for a file in zookeeper was 1MB, my
clusterstate file has 1.1MB. It's possible be this the problem?
If it's, how I can increase this limit?
-----
Best regards
--
View this message in context: http://lucene.472066.n3.nabble.com/clusterstate-json-size-tp4052598.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: clusterstate.json size
Posted by Yago Riveiro <ya...@gmail.com>.
>From Zookeeper documentation:
jute.maxbuffer:
(Java system property: jute.maxbuffer)
This option can only be set as a Java system property. There is no zookeeper prefix on it. It specifies the maximum size of the data that can be stored in a znode. The default is 0xfffff, or just under 1M. If this option is changed, the system property must be set on all servers and clients otherwise problems will arise. This is really a sanity check. ZooKeeper is designed to store data on the order of kilobytes in size.
I can put this on my zookeeper server, but will be work with the SolrZkClient?, the documentation says that must be set in server and clients otherwise problems will arise.
--
Yago Riveiro
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Saturday, March 30, 2013 at 8:00 PM, Yago Riveiro wrote:
> Well, the explanation is very simple. When I started using solr the routing for data was implicit and automatic. Therefore, I did the sharding manually to have control of distribution of data. I wanted data sharding by client and month.
>
> I have 160 clients (and is expected grows) with 7 years of data, sharding by client and month this equals to 13440 (160 * 7 * 12) collections without replication and distributed in 4 nodes.
>
> And I don't have loaded all data yet, and I need to create all replication for collections ...
>
>
> --
> Yago Riveiro
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
>
>
> On Saturday, March 30, 2013 at 6:59 PM, Vaillancourt, Tim wrote:
>
> > Wow. I'm guessing we may have a new "Largest SolrCloud" winner ;).
> >
> > Tim
> >
> > -----Original Message-----
> > From: svambati@gmail.com (mailto:svambati@gmail.com) [mailto:svambati@gmail.com]
> > Sent: Saturday, March 30, 2013 11:56 AM
> > To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> > Cc: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> > Subject: Re: clusterstate.json size
> >
> > You can zip the file and send it to ZK
> >
> > Sent from my iPhone
> >
> > On 30-Mar-2013, at 1:47 PM, Mark Miller <markrmiller@gmail.com (mailto:markrmiller@gmail.com)> wrote:
> >
> > > 4.2.1 *should* return a decent response.
> > >
> > > How many nodes!? I didn't even see that size at 1000 shards!
> > >
> > > It's a zk sys prop to raise it - on the road now, so I'd try google.
> > >
> > > - mark
> > >
> > > Sent from my iPhone
> > >
> > > On Mar 30, 2013, at 1:49 PM, yriveiro <yago.riveiro@gmail.com (mailto:yago.riveiro@gmail.com)> wrote:
> > >
> > > > Hi,
> > > >
> > > > Is there a size limitation for the clusterstate file?
> > > >
> > > > I can't create more collections for my cluster I have no error but
> > > > the CREATE command not return any response.
> > > >
> > > > I read in the past that the max size for a file in zookeeper was 1MB,
> > > > my clusterstate file has 1.1MB. It's possible be this the problem?
> > > >
> > > > If it's, how I can increase this limit?
> > > >
> > > >
> > > >
> > > > -----
> > > > Best regards
> > > > --
> > > > View this message in context:
> > > > http://lucene.472066.n3.nabble.com/clusterstate-json-size-tp4052598.h
> > > > tml Sent from the Solr - User mailing list archive at Nabble.com (http://Nabble.com).
> > > >
> > >
> > >
> >
> >
> >
> >
>
>
Re: clusterstate.json size
Posted by Yago Riveiro <ya...@gmail.com>.
Well, the explanation is very simple. When I started using solr the routing for data was implicit and automatic. Therefore, I did the sharding manually to have control of distribution of data. I wanted data sharding by client and month.
I have 160 clients (and is expected grows) with 7 years of data, sharding by client and month this equals to 13440 (160 * 7 * 12) collections without replication and distributed in 4 nodes.
And I don't have loaded all data yet, and I need to create all replication for collections ...
--
Yago Riveiro
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Saturday, March 30, 2013 at 6:59 PM, Vaillancourt, Tim wrote:
> Wow. I'm guessing we may have a new "Largest SolrCloud" winner ;).
>
> Tim
>
> -----Original Message-----
> From: svambati@gmail.com [mailto:svambati@gmail.com]
> Sent: Saturday, March 30, 2013 11:56 AM
> To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> Cc: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> Subject: Re: clusterstate.json size
>
> You can zip the file and send it to ZK
>
> Sent from my iPhone
>
> On 30-Mar-2013, at 1:47 PM, Mark Miller <markrmiller@gmail.com (mailto:markrmiller@gmail.com)> wrote:
>
> > 4.2.1 *should* return a decent response.
> >
> > How many nodes!? I didn't even see that size at 1000 shards!
> >
> > It's a zk sys prop to raise it - on the road now, so I'd try google.
> >
> > - mark
> >
> > Sent from my iPhone
> >
> > On Mar 30, 2013, at 1:49 PM, yriveiro <yago.riveiro@gmail.com (mailto:yago.riveiro@gmail.com)> wrote:
> >
> > > Hi,
> > >
> > > Is there a size limitation for the clusterstate file?
> > >
> > > I can't create more collections for my cluster I have no error but
> > > the CREATE command not return any response.
> > >
> > > I read in the past that the max size for a file in zookeeper was 1MB,
> > > my clusterstate file has 1.1MB. It's possible be this the problem?
> > >
> > > If it's, how I can increase this limit?
> > >
> > >
> > >
> > > -----
> > > Best regards
> > > --
> > > View this message in context:
> > > http://lucene.472066.n3.nabble.com/clusterstate-json-size-tp4052598.h
> > > tml Sent from the Solr - User mailing list archive at Nabble.com (http://Nabble.com).
> > >
> >
> >
>
>
>
RE: clusterstate.json size
Posted by "Vaillancourt, Tim" <TV...@ea.com>.
Wow. I'm guessing we may have a new "Largest SolrCloud" winner ;).
Tim
-----Original Message-----
From: svambati@gmail.com [mailto:svambati@gmail.com]
Sent: Saturday, March 30, 2013 11:56 AM
To: solr-user@lucene.apache.org
Cc: solr-user@lucene.apache.org
Subject: Re: clusterstate.json size
You can zip the file and send it to ZK
Sent from my iPhone
On 30-Mar-2013, at 1:47 PM, Mark Miller <ma...@gmail.com> wrote:
> 4.2.1 *should* return a decent response.
>
> How many nodes!? I didn't even see that size at 1000 shards!
>
> It's a zk sys prop to raise it - on the road now, so I'd try google.
>
> - mark
>
> Sent from my iPhone
>
> On Mar 30, 2013, at 1:49 PM, yriveiro <ya...@gmail.com> wrote:
>
>> Hi,
>>
>> Is there a size limitation for the clusterstate file?
>>
>> I can't create more collections for my cluster I have no error but
>> the CREATE command not return any response.
>>
>> I read in the past that the max size for a file in zookeeper was 1MB,
>> my clusterstate file has 1.1MB. It's possible be this the problem?
>>
>> If it's, how I can increase this limit?
>>
>>
>>
>> -----
>> Best regards
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/clusterstate-json-size-tp4052598.h
>> tml Sent from the Solr - User mailing list archive at Nabble.com.
Re: clusterstate.json size
Posted by sv...@gmail.com.
You can zip the file and send it to ZK
Sent from my iPhone
On 30-Mar-2013, at 1:47 PM, Mark Miller <ma...@gmail.com> wrote:
> 4.2.1 *should* return a decent response.
>
> How many nodes!? I didn't even see that size at 1000 shards!
>
> It's a zk sys prop to raise it - on the road now, so I'd try google.
>
> - mark
>
> Sent from my iPhone
>
> On Mar 30, 2013, at 1:49 PM, yriveiro <ya...@gmail.com> wrote:
>
>> Hi,
>>
>> Is there a size limitation for the clusterstate file?
>>
>> I can't create more collections for my cluster I have no error but the
>> CREATE command not return any response.
>>
>> I read in the past that the max size for a file in zookeeper was 1MB, my
>> clusterstate file has 1.1MB. It's possible be this the problem?
>>
>> If it's, how I can increase this limit?
>>
>>
>>
>> -----
>> Best regards
>> --
>> View this message in context: http://lucene.472066.n3.nabble.com/clusterstate-json-size-tp4052598.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
Re: clusterstate.json size
Posted by Mark Miller <ma...@gmail.com>.
4.2.1 *should* return a decent response.
How many nodes!? I didn't even see that size at 1000 shards!
It's a zk sys prop to raise it - on the road now, so I'd try google.
- mark
Sent from my iPhone
On Mar 30, 2013, at 1:49 PM, yriveiro <ya...@gmail.com> wrote:
> Hi,
>
> Is there a size limitation for the clusterstate file?
>
> I can't create more collections for my cluster I have no error but the
> CREATE command not return any response.
>
> I read in the past that the max size for a file in zookeeper was 1MB, my
> clusterstate file has 1.1MB. It's possible be this the problem?
>
> If it's, how I can increase this limit?
>
>
>
> -----
> Best regards
> --
> View this message in context: http://lucene.472066.n3.nabble.com/clusterstate-json-size-tp4052598.html
> Sent from the Solr - User mailing list archive at Nabble.com.