Limits of Document Size at SolrCloud and Faced Problems with Large Size of Documents

[Furkan KAMACI <fu...@gmail.com>]

Currently I hafer over 50+ millions documents at my index and as I mentiod
before at another question I have some problems while indexing (jetty EOF
exception) I know that problem may not be about index size but just I want
to learn that is there any limit for document size at Solr that if I exceed
it I can have some problems? I am not talking about the theoretical limit.

What are the maximim index size for folks and what they to handle heavy
index rate when having millions of documents. What tuning strategies they
do?

PS: I have 18 machines, 9 shards, each machine has 48 GB RAM and I use Solr
4.2.1 for my SolrCloud.

Re: Limits of Document Size at SolrCloud and Faced Problems with Large Size of Documents

[Erick Erickson <er...@gmail.com>]

You're probably exceeding the size that your servlet container allows.
This assumes you're using curl or some such. You can change it.
How big is the document and how are you sending it to Solr?

Best,
Erick


On Tue, Sep 17, 2013 at 2:24 PM, Furkan KAMACI <fu...@gmail.com>wrote:

> Currently I hafer over 50+ millions documents at my index and as I mentiod
> before at another question I have some problems while indexing (jetty EOF
> exception) I know that problem may not be about index size but just I want
> to learn that is there any limit for document size at Solr that if I exceed
> it I can have some problems? I am not talking about the theoretical limit.
>
> What are the maximim index size for folks and what they to handle heavy
> index rate when having millions of documents. What tuning strategies they
> do?
>
> PS: I have 18 machines, 9 shards, each machine has 48 GB RAM and I use Solr
> 4.2.1 for my SolrCloud.
>

Re: Limits of Document Size at SolrCloud and Faced Problems with Large Size of Documents

[Erick Erickson <er...@gmail.com>]

Ahhhh, good to know Shawn...

Erick


On Fri, Sep 20, 2013 at 1:04 PM, Shawn Heisey <so...@elyograg.org> wrote:

> On 9/20/2013 12:34 PM, Erick Erickson wrote:
> > You're probably exceeding the size that your servlet container allows.
> > This assumes you're using curl or some such. You can change it.
> > How big is the document and how are you sending it to Solr?
>
> The maximum form size is configurable in Solr, not sure whether that
> change went in for 4.1 or 4.2.  Solr will override what the servlet
> container itself has configured.
>
> In the requestDispatcher section of solrconfig.xml, you can have a
> requestParsers tag.  One of the attributes for that tag can be
> formdataUploadLimitInKB.  The default value for that setting is 2048,
> for a maximum POST size of 2MB.  This should be described in the example
> solrconfig.xml file.
>
> Thanks,
> Shawn
>
>

Re: Limits of Document Size at SolrCloud and Faced Problems with Large Size of Documents

[Shawn Heisey <so...@elyograg.org>]

On 9/20/2013 12:34 PM, Erick Erickson wrote:
> You're probably exceeding the size that your servlet container allows.
> This assumes you're using curl or some such. You can change it.
> How big is the document and how are you sending it to Solr?

The maximum form size is configurable in Solr, not sure whether that
change went in for 4.1 or 4.2.  Solr will override what the servlet
container itself has configured.

In the requestDispatcher section of solrconfig.xml, you can have a
requestParsers tag.  One of the attributes for that tag can be
formdataUploadLimitInKB.  The default value for that setting is 2048,
for a maximum POST size of 2MB.  This should be described in the example
solrconfig.xml file.

Thanks,
Shawn

Re: Limits of Document Size at SolrCloud and Faced Problems with Large Size of Documents

[Erick Erickson <er...@gmail.com>]

You're probably exceeding the size that your servlet container allows.
This assumes you're using curl or some such. You can change it.
How big is the document and how are you sending it to Solr?

Best,
Erick

On Tue, Sep 17, 2013 at 4:28 PM, Otis Gospodnetic
<ot...@gmail.com> wrote:
> Hi
>
> 50m docs across 18 servers 48gb RAM ain't much. I doubt you are hitting any
> limits in lucene or solr.
>
> How heavy is your index rate?
>
> Otis
> Solr & ElasticSearch Support
> http://sematext.com/
> On Sep 17, 2013 5:25 PM, "Furkan KAMACI" <fu...@gmail.com> wrote:
>
>> Currently I hafer over 50+ millions documents at my index and as I mentiod
>> before at another question I have some problems while indexing (jetty EOF
>> exception) I know that problem may not be about index size but just I want
>> to learn that is there any limit for document size at Solr that if I exceed
>> it I can have some problems? I am not talking about the theoretical limit.
>>
>> What are the maximim index size for folks and what they to handle heavy
>> index rate when having millions of documents. What tuning strategies they
>> do?
>>
>> PS: I have 18 machines, 9 shards, each machine has 48 GB RAM and I use Solr
>> 4.2.1 for my SolrCloud.
>>

Re: Limits of Document Size at SolrCloud and Faced Problems with Large Size of Documents

[Otis Gospodnetic <ot...@gmail.com>]

Hi

50m docs across 18 servers 48gb RAM ain't much. I doubt you are hitting any
limits in lucene or solr.

How heavy is your index rate?

Otis
Solr & ElasticSearch Support
http://sematext.com/
On Sep 17, 2013 5:25 PM, "Furkan KAMACI" <fu...@gmail.com> wrote:

> Currently I hafer over 50+ millions documents at my index and as I mentiod
> before at another question I have some problems while indexing (jetty EOF
> exception) I know that problem may not be about index size but just I want
> to learn that is there any limit for document size at Solr that if I exceed
> it I can have some problems? I am not talking about the theoretical limit.
>
> What are the maximim index size for folks and what they to handle heavy
> index rate when having millions of documents. What tuning strategies they
> do?
>
> PS: I have 18 machines, 9 shards, each machine has 48 GB RAM and I use Solr
> 4.2.1 for my SolrCloud.
>