SolrCloud trying to upload documents and shards do not have storage anymore

[Deeksha Sharma <ds...@palamida.com>]

Hi,


I am currently using JSON Index Handler to upload documents to a specific collection on SolrCloud. Now what I need to know is:


If I upload documents to SolrCloud collection and the machines hosting Shards for this collection have no storage left, will Solr reject the commit request?


?

Re: SolrCloud trying to upload documents and shards do not have storage anymore

[Deeksha Sharma <ds...@palamida.com>]

Thanks a ton Erick for the help!

Thanks
Deeksha
________________________________________
From: Erick Erickson <er...@gmail.com>
Sent: Sunday, June 26, 2016 8:58 AM
To: solr-user
Subject: Re: SolrCloud trying to upload documents and shards do not have storage anymore

Yes. you should see errors in your log about out of disk space.

However, you'll have lots of other problems. In the background,
merging segments requires extra disk space. Say seg1 and seg2
are being merged into seg3. First seg1 and seg2 are copied
into seg3, then ser1 and seg2 are deleted.

But if you're near running out of disk space, you won't have enough
room and the merge will fail. In general you need at least as much
free space on your disk as your index occupies. So while Solr/Lucene
will fail to index documents, your system will be unstable long before
you hit that problem. More precisely, I'd expect intermittent failures as,
say, your indexing operation happened to coincide with a segment
merge and failed. But by the time you get around to looking, the amount
of disk occupied by seg3 in the example above will be reclaimed....

Best,
Erick

On Sat, Jun 25, 2016 at 3:07 PM, Deeksha Sharma <ds...@palamida.com> wrote:
> Hi,
>
>
> I am currently using JSON Index Handler to upload documents to a specific collection on SolrCloud. Now what I need to know is:
>
>
> If I upload documents to SolrCloud collection and the machines hosting Shards for this collection have no storage left, will Solr reject the commit request?
>
>
> ?

Re: SolrCloud trying to upload documents and shards do not have storage anymore

[Erick Erickson <er...@gmail.com>]

Yes. you should see errors in your log about out of disk space.

However, you'll have lots of other problems. In the background,
merging segments requires extra disk space. Say seg1 and seg2
are being merged into seg3. First seg1 and seg2 are copied
into seg3, then ser1 and seg2 are deleted.

But if you're near running out of disk space, you won't have enough
room and the merge will fail. In general you need at least as much
free space on your disk as your index occupies. So while Solr/Lucene
will fail to index documents, your system will be unstable long before
you hit that problem. More precisely, I'd expect intermittent failures as,
say, your indexing operation happened to coincide with a segment
merge and failed. But by the time you get around to looking, the amount
of disk occupied by seg3 in the example above will be reclaimed....

Best,
Erick

On Sat, Jun 25, 2016 at 3:07 PM, Deeksha Sharma <ds...@palamida.com> wrote:
> Hi,
>
>
> I am currently using JSON Index Handler to upload documents to a specific collection on SolrCloud. Now what I need to know is:
>
>
> If I upload documents to SolrCloud collection and the machines hosting Shards for this collection have no storage left, will Solr reject the commit request?
>
>
> ?