You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Ning Li <ni...@gmail.com> on 2007/05/02 16:53:49 UTC
Re: [jira] Commented: (LUCENE-847) Factor merge policy out of IndexWriter
On 3/23/07, Steven Parkes (JIRA) <ji...@apache.org> wrote:
> In fact, there a few things here that are fairly subtle/important. The relationship/protocol between the writer and policy is pretty strong. This can be seen in the strawman concurrent merge code where the merge policy holds state and relies on being called from a synchronized writer method. If that goes forward anything like it is, it would argue for tightening that api up some. Chris suggested a way to make the writer<>polcy relationship "atomic". I didn't add the code (yet) but I'm not against it.
What does "atomic" mean here?
I'm thinking, can we move segmentInfos from IndexWriter to merge
policy? This way, IndexWriter is in charge of in-mem part and inserts
and deletes, and merge policy is in charge of disk part and merges.
Then only IndexWriter calls methods in merge policy but not the other
way around. Also, it'll be much easier to support concurrent merges in
a merge policy when it owns segmentInfos.
Just my two cents.
Regards,
Ning
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: [jira] Commented: (LUCENE-847) Factor merge policy out of IndexWriter
Posted by Ning Li <ni...@gmail.com>.
> Having the merge policy own segmentInfos sounds kind of hard to me.
> Among other things, there's a lot of code in IndexWriter for managing
> segmentInfos with regards to transactions. I'm pretty wary of touching
> that code. Is there a way around that?
But conceptually, do you agree it's a good idea for MergePolicy to own
segmentInfos?
I know the code for managing segmentInfos w.r.t transactions is hard.
I had to modify it when coding deleteDocuments for IndexWriter. I
wonder, since exceptions are the rare case, can we simply restore
segmentInfos from the last valid segments file? Hopefully it's simple
thus easy to maintain as well. Opinions?
Ning
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: [jira] Commented: (LUCENE-847) Factor merge policy out of IndexWriter
Posted by Steven Parkes <st...@esseff.org>.
By atomic, I meant that the writer knows the policy and the policy knows
the writer, that it wouldn't be possible for to set a merge policy on a
writer without the merge policy "knowing" that. It's easy enough to
implement with Chris's code (which, I think, makes it a compile error)
or with a simple function for writers to register themselves with the
merge policy when the merge policy is set.
Having the merge policy own segmentInfos sounds kind of hard to me.
Among other things, there's a lot of code in IndexWriter for managing
segmentInfos with regards to transactions. I'm pretty wary of touching
that code. Is there a way around that?
-----Original Message-----
From: Ning Li [mailto:ning.li.li@gmail.com]
Sent: Wednesday, May 02, 2007 7:54 AM
To: java-dev@lucene.apache.org
Subject: Re: [jira] Commented: (LUCENE-847) Factor merge policy out of
IndexWriter
On 3/23/07, Steven Parkes (JIRA) <ji...@apache.org> wrote:
> In fact, there a few things here that are fairly subtle/important. The
relationship/protocol between the writer and policy is pretty strong.
This can be seen in the strawman concurrent merge code where the merge
policy holds state and relies on being called from a synchronized writer
method. If that goes forward anything like it is, it would argue for
tightening that api up some. Chris suggested a way to make the
writer<>polcy relationship "atomic". I didn't add the code (yet) but I'm
not against it.
What does "atomic" mean here?
I'm thinking, can we move segmentInfos from IndexWriter to merge
policy? This way, IndexWriter is in charge of in-mem part and inserts
and deletes, and merge policy is in charge of disk part and merges.
Then only IndexWriter calls methods in merge policy but not the other
way around. Also, it'll be much easier to support concurrent merges in
a merge policy when it owns segmentInfos.
Just my two cents.
Regards,
Ning
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org