You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Darrell Burgan <Da...@infor.com> on 2014/05/07 05:34:01 UTC
Physical Files v. Reported Index Size
Hello all, I'm trying to reconcile what I'm seeing in the file system for a Solr index versus what it is reporting in the UI. Here's what I see in the UI for the index:
https://s3-us-west-2.amazonaws.com/pa-darrell/ui.png
As shown, the index is 74.85 GB in size. However, here is what I see in the data folder of the file system on that server:
https://s3-us-west-2.amazonaws.com/pa-darrell/file-system.png
As shown, it is consuming 109 GB of space. Also note that one of the index folders is 75 GB in size.
My question is why the difference, and whether I can remove some of these index folders to reclaim file system space? Or is there a Solr command to do it (is it as obvious as "Optimize")?
If there a manual I should RTFM about the file structure, please point me to it. :)
Thanks!
Darrell
[Description: Infor]<http://www.infor.com/>
Darrell Burgan | Architect, Sr. Principal, PeopleAnswers
office: 214 445 2172 | mobile: 214 564 4450 | fax: 972 692 5386 | darrell.burgan@infor.com<ma...@infor.com> | http://www.infor.com<http://www.infor.com/>
CONFIDENTIALITY NOTE: This email (including any attachments) is confidential and may be protected by legal privilege. If you are not the intended recipient, be aware that any disclosure, copying, distribution, or use of the information contained herein is prohibited. If you have received this message in error, please notify the sender by replying to this message and then delete this message in its entirety. Thank you for your cooperation.
Re: Physical Files v. Reported Index Size
Posted by Greg Walters <gr...@answers.com>.
See which index directory is actually in use by catting the index.properties file, verify nothing is using the others via lsof and you're safe to delete them.
Thanks,
Greg
On May 6, 2014, at 10:34 PM, Darrell Burgan <Da...@infor.com> wrote:
> Hello all, I’m trying to reconcile what I’m seeing in the file system for a Solr index versus what it is reporting in the UI. Here’s what I see in the UI for the index:
>
> https://s3-us-west-2.amazonaws.com/pa-darrell/ui.png
>
> As shown, the index is 74.85 GB in size. However, here is what I see in the data folder of the file system on that server:
>
> https://s3-us-west-2.amazonaws.com/pa-darrell/file-system.png
>
> As shown, it is consuming 109 GB of space. Also note that one of the index folders is 75 GB in size.
>
> My question is why the difference, and whether I can remove some of these index folders to reclaim file system space? Or is there a Solr command to do it (is it as obvious as “Optimize”)?
>
> If there a manual I should RTFM about the file structure, please point me to it. J
>
> Thanks!
> Darrell
>
>
>
> Darrell Burgan | Architect, Sr. Principal, PeopleAnswers
> office: 214 445 2172 | mobile: 214 564 4450 | fax: 972 692 5386 | darrell.burgan@infor.com | http://www.infor.com
> CONFIDENTIALITY NOTE: This email (including any attachments) is confidential and may be protected by legal privilege. If you are not the intended recipient, be aware that any disclosure, copying, distribution, or use of the information contained herein is prohibited. If you have received this message in error, please notify the sender by replying to this message and then delete this message in its entirety. Thank you for your cooperation.
>
>
RE: Physical Files v. Reported Index Size
Posted by Darrell Burgan <Da...@infor.com>.
Perfect - thanks for the information!
-----Original Message-----
From: Shawn Heisey [mailto:solr@elyograg.org]
Sent: Wednesday, May 07, 2014 7:15 PM
To: solr-user@lucene.apache.org
Subject: Re: Physical Files v. Reported Index Size
On 5/6/2014 9:34 PM, Darrell Burgan wrote:
>
> Hello all, I'm trying to reconcile what I'm seeing in the file system
> for a Solr index versus what it is reporting in the UI. Here's what I
> see in the UI for the index:
>
> https://s3-us-west-2.amazonaws.com/pa-darrell/ui.png
>
> As shown, the index is 74.85 GB in size. However, here is what I see
> in the data folder of the file system on that server:
>
> https://s3-us-west-2.amazonaws.com/pa-darrell/file-system.png
>
> As shown, it is consuming 109 GB of space. Also note that one of the
> index folders is 75 GB in size.
>
The Lucene class that gets the index size only knows about the live index. The transaction log and any other copies of the index are invisible to it.
The additional index copies are probably either from running the backup option on the replication handler, or from replications that didn't work right for some reason. If you were running a newer version of Solr, you would be able to see the index directory in use by the core right on the core overview. See the "Instance" section in the upper right corner of this screenshot from Solr 4.7.2:
https://www.dropbox.com/s/n3c10vj81aqpfjy/core_overview.png
Based on the index size, I think we can figure out which index.nnnnnnnn directory you should NOT delete - the one that's 75GB. You can ask the admin UI to tell you exactly which directory is in use, though. Go to Plugins/Stats, CORE, then open "reader" ... you should end up at a page like the following. The "readerDir" value will have the actual path that you need to keep.
https://www.dropbox.com/s/mzml8pgg1d30h0t/stats-core-searcher.png
Thanks,
Shawn
Re: Physical Files v. Reported Index Size
Posted by Shawn Heisey <so...@elyograg.org>.
On 5/6/2014 9:34 PM, Darrell Burgan wrote:
>
> Hello all, I’m trying to reconcile what I’m seeing in the file system
> for a Solr index versus what it is reporting in the UI. Here’s what I
> see in the UI for the index:
>
> https://s3-us-west-2.amazonaws.com/pa-darrell/ui.png
>
> As shown, the index is 74.85 GB in size. However, here is what I see
> in the data folder of the file system on that server:
>
> https://s3-us-west-2.amazonaws.com/pa-darrell/file-system.png
>
> As shown, it is consuming 109 GB of space. Also note that one of the
> index folders is 75 GB in size.
>
The Lucene class that gets the index size only knows about the live
index. The transaction log and any other copies of the index are
invisible to it.
The additional index copies are probably either from running the backup
option on the replication handler, or from replications that didn't work
right for some reason. If you were running a newer version of Solr, you
would be able to see the index directory in use by the core right on the
core overview. See the "Instance" section in the upper right corner of
this screenshot from Solr 4.7.2:
https://www.dropbox.com/s/n3c10vj81aqpfjy/core_overview.png
Based on the index size, I think we can figure out which index.nnnnnnnn
directory you should NOT delete - the one that's 75GB. You can ask the
admin UI to tell you exactly which directory is in use, though. Go to
Plugins/Stats, CORE, then open "reader" ... you should end up at a page
like the following. The "readerDir" value will have the actual path that
you need to keep.
https://www.dropbox.com/s/mzml8pgg1d30h0t/stats-core-searcher.png
Thanks,
Shawn
Re: Physical Files v. Reported Index Size
Posted by Otis Gospodnetic <ot...@gmail.com>.
Darrell,
Look at the top index.xxxxx directory in your second image. Looks like
that's your index, the same one you see in the Solr UI.
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Tue, May 6, 2014 at 11:34 PM, Darrell Burgan <Da...@infor.com>wrote:
> Hello all, I’m trying to reconcile what I’m seeing in the file system
> for a Solr index versus what it is reporting in the UI. Here’s what I see
> in the UI for the index:
>
>
>
> https://s3-us-west-2.amazonaws.com/pa-darrell/ui.png
>
>
>
> As shown, the index is 74.85 GB in size. However, here is what I see in
> the data folder of the file system on that server:
>
>
>
> https://s3-us-west-2.amazonaws.com/pa-darrell/file-system.png
>
>
>
> As shown, it is consuming 109 GB of space. Also note that one of the index
> folders is 75 GB in size.
>
>
>
> My question is why the difference, and whether I can remove some of these
> index folders to reclaim file system space? Or is there a Solr command to
> do it (is it as obvious as “Optimize”)?
>
>
>
> If there a manual I should RTFM about the file structure, please point me
> to it. J
>
>
>
> Thanks!
>
> Darrell
>
>
>
>
>
> [image: Description: Infor] <http://www.infor.com/>
>
> *Darrell Burgan* | Architect, Sr. Principal, PeopleAnswers
>
> office: 214 445 2172 | mobile: 214 564 4450 | fax: 972 692 5386 |
> darrell.burgan@infor.com | http://www.infor.com
>
> CONFIDENTIALITY NOTE: This email (including any attachments) is
> confidential and may be protected by legal privilege. If you are not the
> intended recipient, be aware that any disclosure, copying, distribution, or
> use of the information contained herein is prohibited. If you have
> received this message in error, please notify the sender by replying to
> this message and then delete this message in its entirety. Thank you for
> your cooperation.
>
>
>
Re: Physical Files v. Reported Index Size
Posted by Upayavira <uv...@odoko.co.uk>.
Perhaps because du reports disk block usage, not total file size?
Upayavira
On Wed, May 7, 2014, at 04:34 AM, Darrell Burgan wrote:
Hello all, I’m trying to reconcile what I’m seeing in the file system
for a Solr index versus what it is reporting in the UI. Here’s what I
see in the UI for the index:
[1]https://s3-us-west-2.amazonaws.com/pa-darrell/ui.png
As shown, the index is 74.85 GB in size. However, here is what I see in
the data folder of the file system on that server:
[2]https://s3-us-west-2.amazonaws.com/pa-darrell/file-system.png
As shown, it is consuming 109 GB of space. Also note that one of the
index folders is 75 GB in size.
My question is why the difference, and whether I can remove some of
these index folders to reclaim file system space? Or is there a Solr
command to do it (is it as obvious as “Optimize”)?
If there a manual I should RTFM about the file structure, please point
me to it. J
Thanks!
Darrell
[3]Description: Infor
Darrell Burgan | Architect, Sr. Principal, PeopleAnswers
office: 214 445 2172 | mobile: 214 564 4450 | fax: 972 692
5386 | [4]darrell.burgan@infor.com | [5]http://www.infor.com
CONFIDENTIALITY NOTE: This email (including any attachments) is
confidential and may be protected by legal privilege. If you are not
the intended recipient, be aware that any disclosure, copying,
distribution, or use of the information contained herein is
prohibited. If you have received this message in error, please notify
the sender by replying to this message and then delete this message in
its entirety. Thank you for your cooperation.
References
1. https://s3-us-west-2.amazonaws.com/pa-darrell/ui.png
2. https://s3-us-west-2.amazonaws.com/pa-darrell/file-system.png
3. http://www.infor.com/
4. mailto:darrell.burgan@infor.com
5. http://www.infor.com/