You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Sharif Shahriar <sh...@m2sys.com> on 2018/07/11 08:05:07 UTC
Number of fields in a solrCloud collection config
Is there any limitation on how many field can be added in a solrcloud
collection configset?
After adding 24,520 fields when I wan't to add new fields, it shows
-"Error persisting managed schema at /configs/*/managed-schema"
-"zkClient has disconnected"
Thank you,
Sharif
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Number of fields in a solrCloud collection config
Posted by Sharif Shahriar <sh...@m2sys.com>.
Thanks a lot Shawn for your reply. I'm using SolrCloud v7.3 and using
schema-less approach. I add documents and new fields automatically added in
managed-schema files. I've tried it several times and it stops at around
13,500 fields.
If I try to add fields using SolrCloud API I can add them, but cannot add
using schemaless approach documents adding.
Regards,
Sharif
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Number of fields in a solrCloud collection config
Posted by Sharif Shahriar <sh...@m2sys.com>.
Hi Erick,
Thank you for your suggestions.
Regards,
Sharif
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Number of fields in a solrCloud collection config
Posted by Erick Erickson <er...@gmail.com>.
As Shawn says, there's no hard limit, but having that many fields
almost always indicates a flawed design.
I'd pretty strongly suggest that you reconsider your design with an
eye towards reducing the field count to
something less than, say, 1,000. That number is fairly arbitrary, and
ideally I'd want it even smaller
but it's not a bad target.
And do note that if you elect to do this, the meta-data will still be
hanging around and you must
create a new collection and re-index from scratch to purge it.
Best,
Erick
On Wed, Jul 11, 2018 at 6:55 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 7/11/2018 2:05 AM, Sharif Shahriar wrote:
>>
>> Is there any limitation on how many field can be added in a solrcloud
>> collection configset?
>> After adding 24,520 fields when I wan't to add new fields, it shows
>> -"Error persisting managed schema at /configs/*/managed-schema"
>> -"zkClient has disconnected"
>
>
> There is no hard limit on the number of fields. It is likely that you are
> running into something else. Without detailed logs it's difficult to know
> for sure what the problem is, but I do have one idea:
>
> The most likely limit that I can think of that you'd be running into is the
> limit on the size of an individual entry in the ZooKeeper database -- your
> schema file is getting too big. By default this limit is right around one
> megabyte. Changing it is possible, but it must be done on all ZK servers
> and every single ZK client (which includes Solr).
>
> https://zookeeper.apache.org/doc/r3.4.12/zookeeperAdmin.html#Experimental+Options%2FFeatures
>
> If that's not the problem, then we will need to see your logs to figure out
> what the problem might be. To properly interpret the log, the exact Solr
> version must be known.
>
> Thanks,
> Shawn
>
Re: Number of fields in a solrCloud collection config
Posted by Shawn Heisey <ap...@elyograg.org>.
On 7/11/2018 2:05 AM, Sharif Shahriar wrote:
> Is there any limitation on how many field can be added in a solrcloud
> collection configset?
> After adding 24,520 fields when I wan't to add new fields, it shows
> -"Error persisting managed schema at /configs/*/managed-schema"
> -"zkClient has disconnected"
There is no hard limit on the number of fields. It is likely that you
are running into something else. Without detailed logs it's difficult
to know for sure what the problem is, but I do have one idea:
The most likely limit that I can think of that you'd be running into is
the limit on the size of an individual entry in the ZooKeeper database
-- your schema file is getting too big. By default this limit is right
around one megabyte. Changing it is possible, but it must be done on
all ZK servers and every single ZK client (which includes Solr).
https://zookeeper.apache.org/doc/r3.4.12/zookeeperAdmin.html#Experimental+Options%2FFeatures
If that's not the problem, then we will need to see your logs to figure
out what the problem might be. To properly interpret the log, the exact
Solr version must be known.
Thanks,
Shawn