You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Jamal, Sarfaraz" <Sa...@VerizonWireless.com.INVALID> on 2016/10/28 19:55:03 UTC
Questions about Disk space Usage
Hi Guys,
I am currently investigating an instance of Solr's Disk space usage and I had a few questions I thought you guys might be able to help answer.
First Question
* There is 30 gb's worth of autosuggest data in the /tmp folder. Each file is half of a gigabyte
Is it safe to delete those files?
Second Question
Also, we notice that at times the disk runs down to only having a few gigabytes available, and then goes back to having more space. (the index file literally grows and then shrinks).
Third Question
Is it also safe to delete the log files?
We run a database indexer on a set interval, perhaps that is relevant to this discussion.
Sas
RE: [E] Re: Questions about Disk space Usage
Posted by "Jamal, Sarfaraz" <Sa...@VerizonWireless.com.INVALID>.
Thank you all for your comments and help -
I kept the last days' worth of files form the /tmp folder and removed the rest - without any problems or difficulties.
Sas
-----Original Message-----
From: Walter underwood [mailto:wunder@wunderwood.org]
Sent: Saturday, October 29, 2016 1:10 PM
To: solr-user@lucene.apache.org
Subject: [E] Re: Questions about Disk space Usage
If it works the way I think it does, an empty segment should take the same amount of time to read in as a full segment, but zero time to write out.
wunder
> On Oct 29, 2016, at 9:21 AM, Erick Erickson <er...@gmail.com> wrote:
>
> I would also expect a totally empty segment to be merged very quickly
> as the percent deleted documents weighs heavily when determining
> whether to merge a segment.... but that's based on principle, not deep
> code knowledge.
>
> Best,
> Erick
>
>> On Fri, Oct 28, 2016 at 6:02 PM, Walter Underwood <wu...@wunderwood.org> wrote:
>> After the merge. That is what merges do, clean up segments.
>>
>> I expect it is very rare for a segment to be 100% deleted docs, so it
>> isn’t worth handling that case.
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>>
>>
>>> On Oct 28, 2016, at 5:54 PM, Alexandre Rafalovitch <ar...@gmail.com> wrote:
>>>
>>> Don't the segment that only has deleted documents just gets dropped?
>>> Or does it get dropped _after_ the merge and therefore still sits
>>> around?
>>>
>>> Regards,
>>> Alex.
>>> ----
>>> Solr Example reading group is starting November 2016, join us at
>>> http://j.mp/SolrERG Newsletter and resources for Solr beginners and
>>> intermediates:
>>> http://www.solr-start.com/
>>>
>>>
>>>> On 29 October 2016 at 08:53, Walter Underwood <wu...@wunderwood.org> wrote:
>>>> It is normal for disk usage to double. Under controlled
>>>> circumstances, it can triple, but that probably won’t happen.
>>>>
>>>> This is the second time today that I’ve sent this information to the list.
>>>>
>>>> It can use nearly 2X the space whenever the largest segment(s) are
>>>> merged, especially if there are only a few smaller segments.
>>>>
>>>> In order to use 3X the space, you need to:
>>>>
>>>> 1. Disable merging.
>>>> 2. Delete all the documents.
>>>> 3. Add all the documents.
>>>> 4. Enable merging.
>>>>
>>>> This causes one complete set of segments that are 100% deletes, one
>>>> set that is 0% deletes, then the merge creates another set that is
>>>> 0% deletes. During the merge, the old files remain while the new
>>>> one is created.
>>>>
>>>> wunder
>>>> Walter Underwood
>>>> wunder@wunderwood.org
>>>> http://observer.wunderwood.org/ (my blog)
>>>>
>>>>
>>>>> On Oct 28, 2016, at 2:41 PM, Alexandre Rafalovitch <ar...@gmail.com> wrote:
>>>>>
>>>>> 2) Is probably a merge operation. Lucene index segments are not
>>>>> rewritable in place, so the merge creates a new file, does
>>>>> everything to it, then switches to it.
>>>>>
>>>>> I remember the number was that the space could temporarily triple
>>>>> (?!?) though that may have been before the tiered merge policy.
>>>>>
>>>>> 3) It should be safe to delete old log files. It is standard log4j stuff.
>>>>>
>>>>> ----
>>>>> Solr Example reading group is starting November 2016, join us at
>>>>> http://j.mp/SolrERG Newsletter and resources for Solr beginners
>>>>> and intermediates:
>>>>> http://www.solr-start.com/
>>>>>
>>>>>
>>>>> On 29 October 2016 at 06:55, Jamal, Sarfaraz
>>>>> <Sa...@verizonwireless.com.invalid> wrote:
>>>>>> Hi Guys,
>>>>>>
>>>>>> I am currently investigating an instance of Solr's Disk space usage and I had a few questions I thought you guys might be able to help answer.
>>>>>>
>>>>>> First Question
>>>>>> * There is 30 gb's worth of autosuggest data in the /tmp folder.
>>>>>> Each file is half of a gigabyte Is it safe to delete those files?
>>>>>>
>>>>>> Second Question
>>>>>> Also, we notice that at times the disk runs down to only having a few gigabytes available, and then goes back to having more space. (the index file literally grows and then shrinks).
>>>>>>
>>>>>> Third Question
>>>>>> Is it also safe to delete the log files?
>>>>>>
>>>>>> We run a database indexer on a set interval, perhaps that is relevant to this discussion.
>>>>>>
>>>>>> Sas
>>>>
>>
Re: Questions about Disk space Usage
Posted by Walter underwood <wu...@wunderwood.org>.
If it works the way I think it does, an empty segment should take the same amount of time to read in as a full segment, but zero time to write out.
wunder
> On Oct 29, 2016, at 9:21 AM, Erick Erickson <er...@gmail.com> wrote:
>
> I would also expect a totally empty segment to be merged very quickly
> as the percent deleted documents weighs heavily when determining
> whether to merge a segment.... but that's based on principle, not deep
> code knowledge.
>
> Best,
> Erick
>
>> On Fri, Oct 28, 2016 at 6:02 PM, Walter Underwood <wu...@wunderwood.org> wrote:
>> After the merge. That is what merges do, clean up segments.
>>
>> I expect it is very rare for a segment to be 100% deleted docs, so it isn’t
>> worth handling that case.
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>>
>>
>>> On Oct 28, 2016, at 5:54 PM, Alexandre Rafalovitch <ar...@gmail.com> wrote:
>>>
>>> Don't the segment that only has deleted documents just gets dropped?
>>> Or does it get dropped _after_ the merge and therefore still sits
>>> around?
>>>
>>> Regards,
>>> Alex.
>>> ----
>>> Solr Example reading group is starting November 2016, join us at
>>> http://j.mp/SolrERG
>>> Newsletter and resources for Solr beginners and intermediates:
>>> http://www.solr-start.com/
>>>
>>>
>>>> On 29 October 2016 at 08:53, Walter Underwood <wu...@wunderwood.org> wrote:
>>>> It is normal for disk usage to double. Under controlled circumstances,
>>>> it can triple, but that probably won’t happen.
>>>>
>>>> This is the second time today that I’ve sent this information to the list.
>>>>
>>>> It can use nearly 2X the space whenever the largest segment(s) are
>>>> merged, especially if there are only a few smaller segments.
>>>>
>>>> In order to use 3X the space, you need to:
>>>>
>>>> 1. Disable merging.
>>>> 2. Delete all the documents.
>>>> 3. Add all the documents.
>>>> 4. Enable merging.
>>>>
>>>> This causes one complete set of segments that are 100% deletes,
>>>> one set that is 0% deletes, then the merge creates another set that
>>>> is 0% deletes. During the merge, the old files remain while the
>>>> new one is created.
>>>>
>>>> wunder
>>>> Walter Underwood
>>>> wunder@wunderwood.org
>>>> http://observer.wunderwood.org/ (my blog)
>>>>
>>>>
>>>>> On Oct 28, 2016, at 2:41 PM, Alexandre Rafalovitch <ar...@gmail.com> wrote:
>>>>>
>>>>> 2) Is probably a merge operation. Lucene index segments are not
>>>>> rewritable in place, so the merge creates a new file, does everything
>>>>> to it, then switches to it.
>>>>>
>>>>> I remember the number was that the space could temporarily triple
>>>>> (?!?) though that may have been before the tiered merge policy.
>>>>>
>>>>> 3) It should be safe to delete old log files. It is standard log4j stuff.
>>>>>
>>>>> ----
>>>>> Solr Example reading group is starting November 2016, join us at
>>>>> http://j.mp/SolrERG
>>>>> Newsletter and resources for Solr beginners and intermediates:
>>>>> http://www.solr-start.com/
>>>>>
>>>>>
>>>>> On 29 October 2016 at 06:55, Jamal, Sarfaraz
>>>>> <Sa...@verizonwireless.com.invalid> wrote:
>>>>>> Hi Guys,
>>>>>>
>>>>>> I am currently investigating an instance of Solr's Disk space usage and I had a few questions I thought you guys might be able to help answer.
>>>>>>
>>>>>> First Question
>>>>>> * There is 30 gb's worth of autosuggest data in the /tmp folder. Each file is half of a gigabyte
>>>>>> Is it safe to delete those files?
>>>>>>
>>>>>> Second Question
>>>>>> Also, we notice that at times the disk runs down to only having a few gigabytes available, and then goes back to having more space. (the index file literally grows and then shrinks).
>>>>>>
>>>>>> Third Question
>>>>>> Is it also safe to delete the log files?
>>>>>>
>>>>>> We run a database indexer on a set interval, perhaps that is relevant to this discussion.
>>>>>>
>>>>>> Sas
>>>>
>>
Re: Questions about Disk space Usage
Posted by Erick Erickson <er...@gmail.com>.
I would also expect a totally empty segment to be merged very quickly
as the percent deleted documents weighs heavily when determining
whether to merge a segment.... but that's based on principle, not deep
code knowledge.
Best,
Erick
On Fri, Oct 28, 2016 at 6:02 PM, Walter Underwood <wu...@wunderwood.org> wrote:
> After the merge. That is what merges do, clean up segments.
>
> I expect it is very rare for a segment to be 100% deleted docs, so it isn’t
> worth handling that case.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
>
>> On Oct 28, 2016, at 5:54 PM, Alexandre Rafalovitch <ar...@gmail.com> wrote:
>>
>> Don't the segment that only has deleted documents just gets dropped?
>> Or does it get dropped _after_ the merge and therefore still sits
>> around?
>>
>> Regards,
>> Alex.
>> ----
>> Solr Example reading group is starting November 2016, join us at
>> http://j.mp/SolrERG
>> Newsletter and resources for Solr beginners and intermediates:
>> http://www.solr-start.com/
>>
>>
>> On 29 October 2016 at 08:53, Walter Underwood <wu...@wunderwood.org> wrote:
>>> It is normal for disk usage to double. Under controlled circumstances,
>>> it can triple, but that probably won’t happen.
>>>
>>> This is the second time today that I’ve sent this information to the list.
>>>
>>> It can use nearly 2X the space whenever the largest segment(s) are
>>> merged, especially if there are only a few smaller segments.
>>>
>>> In order to use 3X the space, you need to:
>>>
>>> 1. Disable merging.
>>> 2. Delete all the documents.
>>> 3. Add all the documents.
>>> 4. Enable merging.
>>>
>>> This causes one complete set of segments that are 100% deletes,
>>> one set that is 0% deletes, then the merge creates another set that
>>> is 0% deletes. During the merge, the old files remain while the
>>> new one is created.
>>>
>>> wunder
>>> Walter Underwood
>>> wunder@wunderwood.org
>>> http://observer.wunderwood.org/ (my blog)
>>>
>>>
>>>> On Oct 28, 2016, at 2:41 PM, Alexandre Rafalovitch <ar...@gmail.com> wrote:
>>>>
>>>> 2) Is probably a merge operation. Lucene index segments are not
>>>> rewritable in place, so the merge creates a new file, does everything
>>>> to it, then switches to it.
>>>>
>>>> I remember the number was that the space could temporarily triple
>>>> (?!?) though that may have been before the tiered merge policy.
>>>>
>>>> 3) It should be safe to delete old log files. It is standard log4j stuff.
>>>>
>>>> ----
>>>> Solr Example reading group is starting November 2016, join us at
>>>> http://j.mp/SolrERG
>>>> Newsletter and resources for Solr beginners and intermediates:
>>>> http://www.solr-start.com/
>>>>
>>>>
>>>> On 29 October 2016 at 06:55, Jamal, Sarfaraz
>>>> <Sa...@verizonwireless.com.invalid> wrote:
>>>>> Hi Guys,
>>>>>
>>>>> I am currently investigating an instance of Solr's Disk space usage and I had a few questions I thought you guys might be able to help answer.
>>>>>
>>>>> First Question
>>>>> * There is 30 gb's worth of autosuggest data in the /tmp folder. Each file is half of a gigabyte
>>>>> Is it safe to delete those files?
>>>>>
>>>>> Second Question
>>>>> Also, we notice that at times the disk runs down to only having a few gigabytes available, and then goes back to having more space. (the index file literally grows and then shrinks).
>>>>>
>>>>> Third Question
>>>>> Is it also safe to delete the log files?
>>>>>
>>>>> We run a database indexer on a set interval, perhaps that is relevant to this discussion.
>>>>>
>>>>> Sas
>>>
>
Re: Questions about Disk space Usage
Posted by Walter Underwood <wu...@wunderwood.org>.
After the merge. That is what merges do, clean up segments.
I expect it is very rare for a segment to be 100% deleted docs, so it isn’t
worth handling that case.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Oct 28, 2016, at 5:54 PM, Alexandre Rafalovitch <ar...@gmail.com> wrote:
>
> Don't the segment that only has deleted documents just gets dropped?
> Or does it get dropped _after_ the merge and therefore still sits
> around?
>
> Regards,
> Alex.
> ----
> Solr Example reading group is starting November 2016, join us at
> http://j.mp/SolrERG
> Newsletter and resources for Solr beginners and intermediates:
> http://www.solr-start.com/
>
>
> On 29 October 2016 at 08:53, Walter Underwood <wu...@wunderwood.org> wrote:
>> It is normal for disk usage to double. Under controlled circumstances,
>> it can triple, but that probably won’t happen.
>>
>> This is the second time today that I’ve sent this information to the list.
>>
>> It can use nearly 2X the space whenever the largest segment(s) are
>> merged, especially if there are only a few smaller segments.
>>
>> In order to use 3X the space, you need to:
>>
>> 1. Disable merging.
>> 2. Delete all the documents.
>> 3. Add all the documents.
>> 4. Enable merging.
>>
>> This causes one complete set of segments that are 100% deletes,
>> one set that is 0% deletes, then the merge creates another set that
>> is 0% deletes. During the merge, the old files remain while the
>> new one is created.
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>>
>>
>>> On Oct 28, 2016, at 2:41 PM, Alexandre Rafalovitch <ar...@gmail.com> wrote:
>>>
>>> 2) Is probably a merge operation. Lucene index segments are not
>>> rewritable in place, so the merge creates a new file, does everything
>>> to it, then switches to it.
>>>
>>> I remember the number was that the space could temporarily triple
>>> (?!?) though that may have been before the tiered merge policy.
>>>
>>> 3) It should be safe to delete old log files. It is standard log4j stuff.
>>>
>>> ----
>>> Solr Example reading group is starting November 2016, join us at
>>> http://j.mp/SolrERG
>>> Newsletter and resources for Solr beginners and intermediates:
>>> http://www.solr-start.com/
>>>
>>>
>>> On 29 October 2016 at 06:55, Jamal, Sarfaraz
>>> <Sa...@verizonwireless.com.invalid> wrote:
>>>> Hi Guys,
>>>>
>>>> I am currently investigating an instance of Solr's Disk space usage and I had a few questions I thought you guys might be able to help answer.
>>>>
>>>> First Question
>>>> * There is 30 gb's worth of autosuggest data in the /tmp folder. Each file is half of a gigabyte
>>>> Is it safe to delete those files?
>>>>
>>>> Second Question
>>>> Also, we notice that at times the disk runs down to only having a few gigabytes available, and then goes back to having more space. (the index file literally grows and then shrinks).
>>>>
>>>> Third Question
>>>> Is it also safe to delete the log files?
>>>>
>>>> We run a database indexer on a set interval, perhaps that is relevant to this discussion.
>>>>
>>>> Sas
>>
Re: Questions about Disk space Usage
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
Don't the segment that only has deleted documents just gets dropped?
Or does it get dropped _after_ the merge and therefore still sits
around?
Regards,
Alex.
----
Solr Example reading group is starting November 2016, join us at
http://j.mp/SolrERG
Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/
On 29 October 2016 at 08:53, Walter Underwood <wu...@wunderwood.org> wrote:
> It is normal for disk usage to double. Under controlled circumstances,
> it can triple, but that probably won’t happen.
>
> This is the second time today that I’ve sent this information to the list.
>
> It can use nearly 2X the space whenever the largest segment(s) are
> merged, especially if there are only a few smaller segments.
>
> In order to use 3X the space, you need to:
>
> 1. Disable merging.
> 2. Delete all the documents.
> 3. Add all the documents.
> 4. Enable merging.
>
> This causes one complete set of segments that are 100% deletes,
> one set that is 0% deletes, then the merge creates another set that
> is 0% deletes. During the merge, the old files remain while the
> new one is created.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
>
>> On Oct 28, 2016, at 2:41 PM, Alexandre Rafalovitch <ar...@gmail.com> wrote:
>>
>> 2) Is probably a merge operation. Lucene index segments are not
>> rewritable in place, so the merge creates a new file, does everything
>> to it, then switches to it.
>>
>> I remember the number was that the space could temporarily triple
>> (?!?) though that may have been before the tiered merge policy.
>>
>> 3) It should be safe to delete old log files. It is standard log4j stuff.
>>
>> ----
>> Solr Example reading group is starting November 2016, join us at
>> http://j.mp/SolrERG
>> Newsletter and resources for Solr beginners and intermediates:
>> http://www.solr-start.com/
>>
>>
>> On 29 October 2016 at 06:55, Jamal, Sarfaraz
>> <Sa...@verizonwireless.com.invalid> wrote:
>>> Hi Guys,
>>>
>>> I am currently investigating an instance of Solr's Disk space usage and I had a few questions I thought you guys might be able to help answer.
>>>
>>> First Question
>>> * There is 30 gb's worth of autosuggest data in the /tmp folder. Each file is half of a gigabyte
>>> Is it safe to delete those files?
>>>
>>> Second Question
>>> Also, we notice that at times the disk runs down to only having a few gigabytes available, and then goes back to having more space. (the index file literally grows and then shrinks).
>>>
>>> Third Question
>>> Is it also safe to delete the log files?
>>>
>>> We run a database indexer on a set interval, perhaps that is relevant to this discussion.
>>>
>>> Sas
>
Re: Questions about Disk space Usage
Posted by Walter Underwood <wu...@wunderwood.org>.
It is normal for disk usage to double. Under controlled circumstances,
it can triple, but that probably won’t happen.
This is the second time today that I’ve sent this information to the list.
It can use nearly 2X the space whenever the largest segment(s) are
merged, especially if there are only a few smaller segments.
In order to use 3X the space, you need to:
1. Disable merging.
2. Delete all the documents.
3. Add all the documents.
4. Enable merging.
This causes one complete set of segments that are 100% deletes,
one set that is 0% deletes, then the merge creates another set that
is 0% deletes. During the merge, the old files remain while the
new one is created.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Oct 28, 2016, at 2:41 PM, Alexandre Rafalovitch <ar...@gmail.com> wrote:
>
> 2) Is probably a merge operation. Lucene index segments are not
> rewritable in place, so the merge creates a new file, does everything
> to it, then switches to it.
>
> I remember the number was that the space could temporarily triple
> (?!?) though that may have been before the tiered merge policy.
>
> 3) It should be safe to delete old log files. It is standard log4j stuff.
>
> ----
> Solr Example reading group is starting November 2016, join us at
> http://j.mp/SolrERG
> Newsletter and resources for Solr beginners and intermediates:
> http://www.solr-start.com/
>
>
> On 29 October 2016 at 06:55, Jamal, Sarfaraz
> <Sa...@verizonwireless.com.invalid> wrote:
>> Hi Guys,
>>
>> I am currently investigating an instance of Solr's Disk space usage and I had a few questions I thought you guys might be able to help answer.
>>
>> First Question
>> * There is 30 gb's worth of autosuggest data in the /tmp folder. Each file is half of a gigabyte
>> Is it safe to delete those files?
>>
>> Second Question
>> Also, we notice that at times the disk runs down to only having a few gigabytes available, and then goes back to having more space. (the index file literally grows and then shrinks).
>>
>> Third Question
>> Is it also safe to delete the log files?
>>
>> We run a database indexer on a set interval, perhaps that is relevant to this discussion.
>>
>> Sas
Re: Questions about Disk space Usage
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
2) Is probably a merge operation. Lucene index segments are not
rewritable in place, so the merge creates a new file, does everything
to it, then switches to it.
I remember the number was that the space could temporarily triple
(?!?) though that may have been before the tiered merge policy.
3) It should be safe to delete old log files. It is standard log4j stuff.
----
Solr Example reading group is starting November 2016, join us at
http://j.mp/SolrERG
Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/
On 29 October 2016 at 06:55, Jamal, Sarfaraz
<Sa...@verizonwireless.com.invalid> wrote:
> Hi Guys,
>
> I am currently investigating an instance of Solr's Disk space usage and I had a few questions I thought you guys might be able to help answer.
>
> First Question
> * There is 30 gb's worth of autosuggest data in the /tmp folder. Each file is half of a gigabyte
> Is it safe to delete those files?
>
> Second Question
> Also, we notice that at times the disk runs down to only having a few gigabytes available, and then goes back to having more space. (the index file literally grows and then shrinks).
>
> Third Question
> Is it also safe to delete the log files?
>
> We run a database indexer on a set interval, perhaps that is relevant to this discussion.
>
> Sas