You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rishi Easwaran <ri...@aol.com> on 2015/05/04 12:53:52 UTC
Re: Multiple index.timestamp directories using up disk space
Thanks for the responses Mark and Ramkumar.
The question I had was, why does Solr need 2 copies at any given time, leading to 2x disk space usage.
Not sure if this information is not published anywhere, and makes HW estimation almost impossible for large scale deployment. Even if the copies are temporary, this becomes really expensive, especially when using SSD in production, when the complex size is over 400TB indexes, running 1000's of solr cloud shards.
If a solr follower has decided that it needs to do replication from leader and capture full copy snapshot. Why can't it delete the old information and replicate from scratch, not requiring more disk space.
Is the concern data loss (a case when both leader and follower lose data)?.
Thanks,
Rishi.
-----Original Message-----
From: Mark Miller <ma...@gmail.com>
To: solr-user <so...@lucene.apache.org>
Sent: Tue, Apr 28, 2015 10:52 am
Subject: Re: Multiple index.timestamp directories using up disk space
If copies of the index are not eventually cleaned up, I'd fill a JIRA
to
address the issue. Those directories should be removed over time. At
times
there will have to be a couple around at the same time and others may
take
a while to clean up.
- Mark
On Tue, Apr 28, 2015 at 3:27 AM Ramkumar
R. Aiyengar <
andyetitmoves@gmail.com> wrote:
> SolrCloud does need up to
twice the amount of disk space as your usual
> index size during replication.
Amongst other things, this ensures you have
> a full copy of the index at any
point. There's no way around this, I would
> suggest you provision the
additional disk space needed.
> On 20 Apr 2015 23:21, "Rishi Easwaran"
<ri...@aol.com> wrote:
>
> > Hi All,
> >
> > We are seeing this
problem with solr 4.6 and solr 4.10.3.
> > For some reason, solr cloud tries to
recover and creates a new index
> > directory - (ex:index.20150420181214550),
while keeping the older index
> as
> > is. This creates an issues where the
disk space fills up and the shard
> > never ends up recovering.
> > Usually
this requires a manual intervention of bouncing the instance and
> > wiping
the disk clean to allow for a clean recovery.
> >
> > Any ideas on how to
prevent solr from creating multiple copies of index
> > directory.
> >
> >
Thanks,
> > Rishi.
> >
>
Re: Multiple index.timestamp directories using up disk space
Posted by Rishi Easwaran <ri...@aol.com>.
Walter,
Unless I am missing something here.. I completely get that, when a few segment merges solr requires 2x space of segments to accomplish this.
Usually any index has multiple segments files so this fragmented 2x space consumption is not an issue, even as merged segments grow bigger.
But what I am talking about is copy of a whole index as is into a new directory. The new directory has no relation to the older index directory or its segments, so not sure what merges are going on across directories/indexes, and why solr needs the older index.
Thanks,
Rishi.
-----Original Message-----
From: Walter Underwood <wu...@wunderwood.org>
To: solr-user <so...@lucene.apache.org>
Sent: Mon, May 4, 2015 9:50 am
Subject: Re: Multiple index.timestamp directories using up disk space
One segment is in-use, being searched. That segment (and others) are merged into
a new segment. After the new segment is ready, searches are directed to the new
copy and the old copies are deleted.
That is how two copies are needed.
If
you cannot provide 2X the disk space, you will not have a stable Solr
installation. You should consider a different search engine.
“Optimizing”
(forced merges) will not help. It will probably cause failures more often
because it always merges the larges segment.
Walter
Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my
blog)
On May 4, 2015, at 3:53 AM, Rishi Easwaran <ri...@aol.com>
wrote:
> Thanks for the responses Mark and Ramkumar.
>
> The question I
had was, why does Solr need 2 copies at any given time, leading to 2x disk space
usage.
> Not sure if this information is not published anywhere, and makes HW
estimation almost impossible for large scale deployment. Even if the copies are
temporary, this becomes really expensive, especially when using SSD in
production, when the complex size is over 400TB indexes, running 1000's of solr
cloud shards.
>
> If a solr follower has decided that it needs to do
replication from leader and capture full copy snapshot. Why can't it delete the
old information and replicate from scratch, not requiring more disk space.
> Is
the concern data loss (a case when both leader and follower lose data)?.
>
>
Thanks,
> Rishi.
>
>
>
>
>
>
>
> -----Original
Message-----
> From: Mark Miller <ma...@gmail.com>
> To: solr-user
<so...@lucene.apache.org>
> Sent: Tue, Apr 28, 2015 10:52 am
> Subject:
Re: Multiple index.timestamp directories using up disk space
>
>
> If
copies of the index are not eventually cleaned up, I'd fill a JIRA
> to
>
address the issue. Those directories should be removed over time. At
> times
>
there will have to be a couple around at the same time and others may
> take
>
a while to clean up.
>
> - Mark
>
> On Tue, Apr 28, 2015 at 3:27 AM
Ramkumar
> R. Aiyengar <
> andyetitmoves@gmail.com> wrote:
>
>> SolrCloud
does need up to
> twice the amount of disk space as your usual
>> index size
during replication.
> Amongst other things, this ensures you have
>> a full
copy of the index at any
> point. There's no way around this, I would
>>
suggest you provision the
> additional disk space needed.
>> On 20 Apr 2015
23:21, "Rishi Easwaran"
> <ri...@aol.com> wrote:
>>
>>> Hi
All,
>>>
>>> We are seeing this
> problem with solr 4.6 and solr
4.10.3.
>>> For some reason, solr cloud tries to
> recover and creates a new
index
>>> directory - (ex:index.20150420181214550),
> while keeping the older
index
>> as
>>> is. This creates an issues where the
> disk space fills up
and the shard
>>> never ends up recovering.
>>> Usually
> this requires a
manual intervention of bouncing the instance and
>>> wiping
> the disk clean
to allow for a clean recovery.
>>>
>>> Any ideas on how to
> prevent solr
from creating multiple copies of index
>>> directory.
>>>
>>>
>
Thanks,
>>> Rishi.
>>>
>>
>
>
Re: Multiple index.timestamp directories using up disk space
Posted by Walter Underwood <wu...@wunderwood.org>.
One segment is in-use, being searched. That segment (and others) are merged into a new segment. After the new segment is ready, searches are directed to the new copy and the old copies are deleted.
That is how two copies are needed.
If you cannot provide 2X the disk space, you will not have a stable Solr installation. You should consider a different search engine.
“Optimizing” (forced merges) will not help. It will probably cause failures more often because it always merges the larges segment.
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
On May 4, 2015, at 3:53 AM, Rishi Easwaran <ri...@aol.com> wrote:
> Thanks for the responses Mark and Ramkumar.
>
> The question I had was, why does Solr need 2 copies at any given time, leading to 2x disk space usage.
> Not sure if this information is not published anywhere, and makes HW estimation almost impossible for large scale deployment. Even if the copies are temporary, this becomes really expensive, especially when using SSD in production, when the complex size is over 400TB indexes, running 1000's of solr cloud shards.
>
> If a solr follower has decided that it needs to do replication from leader and capture full copy snapshot. Why can't it delete the old information and replicate from scratch, not requiring more disk space.
> Is the concern data loss (a case when both leader and follower lose data)?.
>
> Thanks,
> Rishi.
>
>
>
>
>
>
>
> -----Original Message-----
> From: Mark Miller <ma...@gmail.com>
> To: solr-user <so...@lucene.apache.org>
> Sent: Tue, Apr 28, 2015 10:52 am
> Subject: Re: Multiple index.timestamp directories using up disk space
>
>
> If copies of the index are not eventually cleaned up, I'd fill a JIRA
> to
> address the issue. Those directories should be removed over time. At
> times
> there will have to be a couple around at the same time and others may
> take
> a while to clean up.
>
> - Mark
>
> On Tue, Apr 28, 2015 at 3:27 AM Ramkumar
> R. Aiyengar <
> andyetitmoves@gmail.com> wrote:
>
>> SolrCloud does need up to
> twice the amount of disk space as your usual
>> index size during replication.
> Amongst other things, this ensures you have
>> a full copy of the index at any
> point. There's no way around this, I would
>> suggest you provision the
> additional disk space needed.
>> On 20 Apr 2015 23:21, "Rishi Easwaran"
> <ri...@aol.com> wrote:
>>
>>> Hi All,
>>>
>>> We are seeing this
> problem with solr 4.6 and solr 4.10.3.
>>> For some reason, solr cloud tries to
> recover and creates a new index
>>> directory - (ex:index.20150420181214550),
> while keeping the older index
>> as
>>> is. This creates an issues where the
> disk space fills up and the shard
>>> never ends up recovering.
>>> Usually
> this requires a manual intervention of bouncing the instance and
>>> wiping
> the disk clean to allow for a clean recovery.
>>>
>>> Any ideas on how to
> prevent solr from creating multiple copies of index
>>> directory.
>>>
>>>
> Thanks,
>>> Rishi.
>>>
>>
>
>
Re: Multiple index.timestamp directories using up disk space
Posted by rishi <ri...@aol.com>.
We use the following merge policy on SSD's and are running on physical
machines with linux OS.
<mergeFactor>10</mergeFactor>
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy"/>
<mergeScheduler
class="org.apache.lucene.index.ConcurrentMergeScheduler">
<int name="maxThreadCount">3</int>
<int name="maxMergeCount">15</int>
</mergeScheduler>
<ramBufferSizeMB>64</ramBufferSizeMB>
Not sure if its very aggressive, but its something we keep to prevent
deleted documents taking up too much space on our index.
Is there some error message that solr logs when rename and deletion of the
directories fails. If so we could monitor our logs to get a better idea for
the root cause. At present we can only react when things go wrong based on
disk space alarms.
Thanks,
Rishi.
--
View this message in context: http://lucene.472066.n3.nabble.com/Multiple-index-timestamp-directories-using-up-disk-space-tp4201098p4204145.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Multiple index.timestamp directories using up disk space
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/5/2015 1:15 PM, Rishi Easwaran wrote:
> Thanks for clarifying lucene segment behaviour. We don't trigger optimize externally, could it be internal solr optimize? Is there a setting/ knob to control when optimize occurs.
Optimize never happens automatically, but *merging* does. An optimize
is nothing more than a forced merge down to one segment. There is a
merge policy, consulted anytime a new segment is created, that decides
whether any automatic merges need to take place and what segments will
be merged. That merge policy can be configured in solrconfig.xml.
> The behaviour we see multiple huge directories for the same core. Till we figure out what's going on, the only option we are left with it is to clean up the entire index to free up disk space, and allow a replica to sync from scratch.
If multiple index directories exist after replication, there was either
a problem that prevented the rename and deletion of the directories
(common on Windows, less common on UNIX variants like Linux), or you're
running into a bug. Unless you are performing maintenance or a machine
goes down, index recovery (replication) should *not* be happening during
normal operation of a SolrCloud cluster. Frequent index recoveries
usually mean that there's a performance problem.
Solr performs better on bare metal than on virtual machines.
Thanks,
Shawn
Re: Multiple index.timestamp directories using up disk space
Posted by Rishi Easwaran <ri...@aol.com>.
Hi Shawn,
Thanks for clarifying lucene segment behaviour. We don't trigger optimize externally, could it be internal solr optimize? Is there a setting/ knob to control when optimize occurs.
Thanks for pointing it out, will monitor memory closely. Though doubt memory is an issue, these are top tier machines with 144GB RAM supporting 12x4GB JVM's. Out of which 9 JVM's are running in cloud mode writing to SSD, should be enough memory leftover for OS cache.
The behaviour we see multiple huge directories for the same core. Till we figure out what's going on, the only option we are left with it is to clean up the entire index to free up disk space, and allow a replica to sync from scratch.
Thanks,
Rishi.
-----Original Message-----
From: Shawn Heisey <ap...@elyograg.org>
To: solr-user <so...@lucene.apache.org>
Sent: Tue, May 5, 2015 10:55 am
Subject: Re: Multiple index.timestamp directories using up disk space
On 5/5/2015 7:29 AM, Rishi Easwaran wrote:
> Worried about data loss makes
sense. If I get the way solr behaves, the new directory should only have
missing/changed segments.
> I guess since our application is extremely write
heavy, with lot of inserts and deletes, almost every segment is touched even
during a short window, so it appears like for our deployment every segment is
copied over when replicas get out of sync.
Once a segment is written, it is
*NEVER* updated again. This aspect of
Lucene indexes makes Solr replication
more efficient. The ids of
deleted documents are written to separate files
specifically for
tracking deletes. Those files are typically quite small
compared to the
index segments. Any new documents are inserted into new
segments.
When older segments are merged, the information in all of those
segments
is copied to a single new segment (minus documents marked as
deleted),
and then the old segments are erased. Optimizing replaces the
entire
index, and each replica of the index would be considered different,
so
an index recovery that happens after optimization might copy the
whole
thing.
If you are seeing a lot of index recoveries during normal
operation,
chances are that your Solr servers do not have enough resources, and
the
resource that has the most impact on performance is memory. The amount
of
memory required for good Solr performance is higher than most people
expect.
It's a normal expectation that programs require memory to run,
but Solr has an
additional memory requirement that often surprises them
-- the need for a
significant OS disk
cache:
http://wiki.apache.org/solr/SolrPerformanceProblems
Thanks,
Shawn
Re: Multiple index.timestamp directories using up disk space
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/5/2015 7:29 AM, Rishi Easwaran wrote:
> Worried about data loss makes sense. If I get the way solr behaves, the new directory should only have missing/changed segments.
> I guess since our application is extremely write heavy, with lot of inserts and deletes, almost every segment is touched even during a short window, so it appears like for our deployment every segment is copied over when replicas get out of sync.
Once a segment is written, it is *NEVER* updated again. This aspect of
Lucene indexes makes Solr replication more efficient. The ids of
deleted documents are written to separate files specifically for
tracking deletes. Those files are typically quite small compared to the
index segments. Any new documents are inserted into new segments.
When older segments are merged, the information in all of those segments
is copied to a single new segment (minus documents marked as deleted),
and then the old segments are erased. Optimizing replaces the entire
index, and each replica of the index would be considered different, so
an index recovery that happens after optimization might copy the whole
thing.
If you are seeing a lot of index recoveries during normal operation,
chances are that your Solr servers do not have enough resources, and the
resource that has the most impact on performance is memory. The amount
of memory required for good Solr performance is higher than most people
expect. It's a normal expectation that programs require memory to run,
but Solr has an additional memory requirement that often surprises them
-- the need for a significant OS disk cache:
http://wiki.apache.org/solr/SolrPerformanceProblems
Thanks,
Shawn
Re: Multiple index.timestamp directories using up disk space
Posted by Rishi Easwaran <ri...@aol.com>.
Worried about data loss makes sense. If I get the way solr behaves, the new directory should only have missing/changed segments.
I guess since our application is extremely write heavy, with lot of inserts and deletes, almost every segment is touched even during a short window, so it appears like for our deployment every segment is copied over when replicas get out of sync.
Thanks for clarifying this behaviour from solr cloud so we can put in external steps to resolve when this situation arises.
-----Original Message-----
From: Ramkumar R. Aiyengar <an...@gmail.com>
To: solr-user <so...@lucene.apache.org>
Sent: Tue, May 5, 2015 4:52 am
Subject: Re: Multiple index.timestamp directories using up disk space
Yes, data loss is the concern. If the recovering replica is not able
to
retrieve the files from the leader, it at least has an older copy.
Also,
the entire index is not fetched from the leader, only the segments
which have
changed. The replica initially gets the file list from the
replica, checks
against what it has, and then downloads the difference --
then moves it to the
main index. Note that this process can fail sometimes
(say due to I/O errors,
or due to a problem with the leader itself), in
which case the replica drops
all accumulated files from the leader, and
starts from scratch. If that
happens, it needs to look back at its old
index again to figure out what it
needs to download on the next attempt.
May be with a fair number of
assumptions which should usually hold good,
you can still come up with a
mechanism to drop existing files, but those
won't hold good in case of serious
issues with the cloud, you could end up
losing data. That's worse than using a
bit more disk space!
On 4 May 2015 11:56, "Rishi Easwaran"
<ri...@aol.com> wrote:
Thanks for the responses Mark and
Ramkumar.
The question I had was, why does Solr need 2 copies at any given
time,
leading to 2x disk space usage.
Not sure if this information is not
published anywhere, and makes HW
estimation almost impossible for large scale
deployment. Even if the copies
are temporary, this becomes really expensive,
especially when using SSD in
production, when the complex size is over 400TB
indexes, running 1000's of
solr cloud shards.
If a solr follower has
decided that it needs to do replication from leader
and capture full copy
snapshot. Why can't it delete the old information and
replicate from scratch,
not requiring more disk space.
Is the concern data loss (a case when both
leader and follower lose data)?.
Thanks,
Rishi.
-----Original Message-----
From: Mark Miller
<ma...@gmail.com>
To: solr-user <so...@lucene.apache.org>
Sent: Tue,
Apr 28, 2015 10:52 am
Subject: Re: Multiple index.timestamp directories using
up disk space
If copies of the index are not eventually cleaned up, I'd
fill a JIRA
to
address the issue. Those directories should be removed over
time. At
times
there will have to be a couple around at the same time and
others may
take
a while to clean up.
- Mark
On Tue, Apr 28, 2015 at 3:27
AM Ramkumar
R. Aiyengar <
andyetitmoves@gmail.com> wrote:
> SolrCloud does
need up to
twice the amount of disk space as your usual
> index size during
replication.
Amongst other things, this ensures you have
> a full copy of the
index at any
point. There's no way around this, I would
> suggest you
provision the
additional disk space needed.
> On 20 Apr 2015 23:21, "Rishi
Easwaran"
<ri...@aol.com> wrote:
>
> > Hi All,
> >
> > We are
seeing this
problem with solr 4.6 and solr 4.10.3.
> > For some reason, solr
cloud tries to
recover and creates a new index
> > directory -
(ex:index.20150420181214550),
while keeping the older index
> as
> > is. This
creates an issues where the
disk space fills up and the shard
> > never ends
up recovering.
> > Usually
this requires a manual intervention of bouncing
the instance and
> > wiping
the disk clean to allow for a clean recovery.
>
>
> > Any ideas on how to
prevent solr from creating multiple copies of
index
> > directory.
> >
> >
Thanks,
> > Rishi.
> >
>
Re: Multiple index.timestamp directories using up disk space
Posted by "Ramkumar R. Aiyengar" <an...@gmail.com>.
Yes, data loss is the concern. If the recovering replica is not able to
retrieve the files from the leader, it at least has an older copy.
Also, the entire index is not fetched from the leader, only the segments
which have changed. The replica initially gets the file list from the
replica, checks against what it has, and then downloads the difference --
then moves it to the main index. Note that this process can fail sometimes
(say due to I/O errors, or due to a problem with the leader itself), in
which case the replica drops all accumulated files from the leader, and
starts from scratch. If that happens, it needs to look back at its old
index again to figure out what it needs to download on the next attempt.
May be with a fair number of assumptions which should usually hold good,
you can still come up with a mechanism to drop existing files, but those
won't hold good in case of serious issues with the cloud, you could end up
losing data. That's worse than using a bit more disk space!
On 4 May 2015 11:56, "Rishi Easwaran" <ri...@aol.com> wrote:
Thanks for the responses Mark and Ramkumar.
The question I had was, why does Solr need 2 copies at any given time,
leading to 2x disk space usage.
Not sure if this information is not published anywhere, and makes HW
estimation almost impossible for large scale deployment. Even if the copies
are temporary, this becomes really expensive, especially when using SSD in
production, when the complex size is over 400TB indexes, running 1000's of
solr cloud shards.
If a solr follower has decided that it needs to do replication from leader
and capture full copy snapshot. Why can't it delete the old information and
replicate from scratch, not requiring more disk space.
Is the concern data loss (a case when both leader and follower lose data)?.
Thanks,
Rishi.
-----Original Message-----
From: Mark Miller <ma...@gmail.com>
To: solr-user <so...@lucene.apache.org>
Sent: Tue, Apr 28, 2015 10:52 am
Subject: Re: Multiple index.timestamp directories using up disk space
If copies of the index are not eventually cleaned up, I'd fill a JIRA
to
address the issue. Those directories should be removed over time. At
times
there will have to be a couple around at the same time and others may
take
a while to clean up.
- Mark
On Tue, Apr 28, 2015 at 3:27 AM Ramkumar
R. Aiyengar <
andyetitmoves@gmail.com> wrote:
> SolrCloud does need up to
twice the amount of disk space as your usual
> index size during replication.
Amongst other things, this ensures you have
> a full copy of the index at any
point. There's no way around this, I would
> suggest you provision the
additional disk space needed.
> On 20 Apr 2015 23:21, "Rishi Easwaran"
<ri...@aol.com> wrote:
>
> > Hi All,
> >
> > We are seeing this
problem with solr 4.6 and solr 4.10.3.
> > For some reason, solr cloud tries to
recover and creates a new index
> > directory - (ex:index.20150420181214550),
while keeping the older index
> as
> > is. This creates an issues where the
disk space fills up and the shard
> > never ends up recovering.
> > Usually
this requires a manual intervention of bouncing the instance and
> > wiping
the disk clean to allow for a clean recovery.
> >
> > Any ideas on how to
prevent solr from creating multiple copies of index
> > directory.
> >
> >
Thanks,
> > Rishi.
> >
>