You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucenenet.apache.org by Floyd Wu <fl...@gmail.com> on 2010/03/08 10:44:54 UTC
About processing handling during merge time
Hi all,
I have a problem that my index files is 33.4gb now, and when I insert new
index through Lucene.Net few times. Lucene will execute merge index files
automatically. The problem is during merge executed, writer.lock file will
be created and all operations such as read/write index files were all
stopped.
(for a long time and there were no any status or flag to know Lucene were
executeing merge process.)
Are there any solutions to avoid this "long time lock" situation?
Thanks in advanced.
Floyd
RE: About processing handling during merge time
Posted by Michael Garski <mg...@myspace-inc.com>.
Floyd,
Implementing a custom merge policy involves a concrete implementation
Lucene.Net.Index.MergePolicy. The only documentation to point you to
are the javadocs as the API between Java Lucene & Lucene.Net are
identical:
http://lucene.apache.org/java/2_9_2/api/all/org/apache/lucene/index/Merg
ePolicy.html
For an implementation example take a look at
Lucene.Net.Index.LogMergePolicy.
Michael
-----Original Message-----
From: Floyd Wu [mailto:floyd.wu@gmail.com]
Sent: Tuesday, March 09, 2010 7:14 PM
To: lucene-net-user@lucene.apache.org
Subject: Re: About processing handling during merge time
Hi Michael,
If I want to create a custom merge policy that act "Never do merge
during
work day". Where to start? I read something about merge policy could be
related with scheduler, but I'm not sure the relationship between these
two
objects.
many thanks.
Floyd
2010/3/9 Michael Garski <mg...@myspace-inc.com>
> Floyd,
>
> Long-running merges are an issue with large indexes due to the CPU and
> I/O overhead required to perform the merge, but there are a few things
> you can do to reduce that time:
>
> 1) Create a custom merge policy to give you control over when merges
> occur and what segments are merged
> 2) Shard your index into smaller pieces that are searched together
using
> a MultiSearcher - this will keep your indexes smaller.
>
> Michael
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Monday, March 08, 2010 1:45 AM
> To: lucene-net-user@incubator.apache.org
> Subject: About processing handling during merge time
>
> Hi all,
>
> I have a problem that my index files is 33.4gb now, and when I insert
> new
> index through Lucene.Net few times. Lucene will execute merge index
> files
> automatically. The problem is during merge executed, writer.lock file
> will
> be created and all operations such as read/write index files were all
> stopped.
> (for a long time and there were no any status or flag to know Lucene
> were
> executeing merge process.)
>
> Are there any solutions to avoid this "long time lock" situation?
>
> Thanks in advanced.
>
> Floyd
>
>
Re: About processing handling during merge time
Posted by Floyd Wu <fl...@gmail.com>.
Hi Michael,
If I want to create a custom merge policy that act "Never do merge during
work day". Where to start? I read something about merge policy could be
related with scheduler, but I'm not sure the relationship between these two
objects.
many thanks.
Floyd
2010/3/9 Michael Garski <mg...@myspace-inc.com>
> Floyd,
>
> Long-running merges are an issue with large indexes due to the CPU and
> I/O overhead required to perform the merge, but there are a few things
> you can do to reduce that time:
>
> 1) Create a custom merge policy to give you control over when merges
> occur and what segments are merged
> 2) Shard your index into smaller pieces that are searched together using
> a MultiSearcher - this will keep your indexes smaller.
>
> Michael
>
> -----Original Message-----
> From: Floyd Wu [mailto:floyd.wu@gmail.com]
> Sent: Monday, March 08, 2010 1:45 AM
> To: lucene-net-user@incubator.apache.org
> Subject: About processing handling during merge time
>
> Hi all,
>
> I have a problem that my index files is 33.4gb now, and when I insert
> new
> index through Lucene.Net few times. Lucene will execute merge index
> files
> automatically. The problem is during merge executed, writer.lock file
> will
> be created and all operations such as read/write index files were all
> stopped.
> (for a long time and there were no any status or flag to know Lucene
> were
> executeing merge process.)
>
> Are there any solutions to avoid this "long time lock" situation?
>
> Thanks in advanced.
>
> Floyd
>
>
RE: About processing handling during merge time
Posted by Michael Garski <mg...@myspace-inc.com>.
Floyd,
Long-running merges are an issue with large indexes due to the CPU and
I/O overhead required to perform the merge, but there are a few things
you can do to reduce that time:
1) Create a custom merge policy to give you control over when merges
occur and what segments are merged
2) Shard your index into smaller pieces that are searched together using
a MultiSearcher - this will keep your indexes smaller.
Michael
-----Original Message-----
From: Floyd Wu [mailto:floyd.wu@gmail.com]
Sent: Monday, March 08, 2010 1:45 AM
To: lucene-net-user@incubator.apache.org
Subject: About processing handling during merge time
Hi all,
I have a problem that my index files is 33.4gb now, and when I insert
new
index through Lucene.Net few times. Lucene will execute merge index
files
automatically. The problem is during merge executed, writer.lock file
will
be created and all operations such as read/write index files were all
stopped.
(for a long time and there were no any status or flag to know Lucene
were
executeing merge process.)
Are there any solutions to avoid this "long time lock" situation?
Thanks in advanced.
Floyd