You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Chuck Williams (JIRA)" <ji...@apache.org> on 2006/06/12 22:49:32 UTC
[jira] Updated: (LUCENE-600) ParallelWriter companion to
ParallelReader
[ http://issues.apache.org/jira/browse/LUCENE-600?page=all ]
Chuck Williams updated LUCENE-600:
----------------------------------
Attachment: ParallelWriter.patch
Patch to create and integrate ParallelWriter, Writable and TestParallelWriter -- also modifies build to use java 1.5.
> ParallelWriter companion to ParallelReader
> ------------------------------------------
>
> Key: LUCENE-600
> URL: http://issues.apache.org/jira/browse/LUCENE-600
> Project: Lucene - Java
> Type: Improvement
> Components: Index
> Versions: 2.1
> Reporter: Chuck Williams
> Attachments: ParallelWriter.patch
>
> A new class ParallelWriter is provided that serves as a companion to ParallelReader. ParallelWriter meets all of the doc-id synchronization requirements of ParallelReader, subject to:
> 1. ParallelWriter.addDocument() is synchronized, which might have an adverse effect on performance. The writes to the sub-indexes are, however, done in parallel.
> 2. The application must ensure that the ParallelReader is never reopened inside ParallelWriter.addDocument(), else it might find the sub-indexes out of sync.
> 3. The application must deal with recovery from ParallelWriter.addDocument() exceptions. Recovery must restore the synchronization of doc-ids, e.g. by deleting any trailing document(s) in one sub-index that were not successfully added to all sub-indexes, and then optimizing all sub-indexes.
> A new interface, Writable, is provided to abstract IndexWriter and ParallelWriter. This is in the same spirit as the existing Searchable and Fieldable classes.
> This implementation uses java 1.5. The patch applies against today's svn head. All tests pass, including the new TestParallelWriter.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Java 1.5
Posted by Doug Cutting <cu...@apache.org>.
Chuck Williams wrote:
> I think the last discussion ended with the main counter-argument being
> lack of support by gjc. Current top of GJC News:
>
>> *June 6, 2006* RMS approved the plan to use the Eclipse compiler as
>> the new gcj front end. Work is being done on the |gcj-eclipse| branch;
>> it can already build libgcj. This project will allow us to ship a 1.5
>> compiler in the relatively near future. The old |gcjx| branch and
>> project is now dead.
Another Java implementation to track is Apache Harmony. Harmony intends
to have a 1.5-compatible JVM completed by Q1 2007.
http://incubator.apache.org/harmony/roadmap.html
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Java 1.5 was [jira] Updated: (LUCENE-600) ParallelWriter companion to ParallelReader
Posted by Otis Gospodnetic <ot...@yahoo.com>.
I'll just send it to java-user in a bit in order to get the answers only from Lucene users (and not peeps just passing by lucene.apache.org).
Otis
----- Original Message ----
From: Grant Ingersoll <gs...@syr.edu>
To: java-dev@lucene.apache.org
Sent: Friday, June 16, 2006 6:53:57 AM
Subject: Re: Java 1.5 was [jira] Updated: (LUCENE-600) ParallelWriter companion to ParallelReader
+1
Do you want to post it on the user list? It might also be good to put
it up on the main website.
Otis Gospodnetic wrote:
> Grant: how to poll users? How about this: http://www.quimble.com/poll/view/2156 ? If you think that's ok, we can send that to java-user tomorrow and see. Hey, how about some bets? I'll put a $10 for a beer on 1.5.
>
>
Wow, $10 for a beer? That must be some pretty good beer. Either that
or you live in New York City and that is a cheap beer! Anyway, I am
betting it is 1.5 as well. Maybe we can get together at ApacheCon or
something for one...
> Otis
>
> ----- Original Message ----
> From: Grant Ingersoll <gs...@syr.edu>
> To: java-dev@lucene.apache.org
> Sent: Tuesday, June 13, 2006 5:01:30 PM
> Subject: Re: Java 1.5 was [jira] Updated: (LUCENE-600) ParallelWriter companion to ParallelReader
>
>
>
>> In addition to performance, productivity and functionality benefits, my
>> main argument for 1.5 is that it is used by the vast majority of lucene
>> community members.
>>
>
> I am not so sure about this. Perhaps we should take a poll on the user
> list? Not even sure how that would be managed or counted, but...
>
>
>> Everything I write is in 1.5 and I don't have time
>> to backport. I have a significant body of code from which to extract
>> and contribute patches that others would likely find useful. How many
>> others are in a similar position?
>>
>>
> I definitely would prefer to make future contributions in 1.5 (even the
> patch we just contributed (issue 545) could have been better given 1.5,
> but it is fine with 1.4 as well). I tend to think if people don't want
> the new functionality or if it breaks their app. then they need not
> upgrade, or they can contribute patches against the branches for prior
> releases and we can support that as needed. To me, this is what major
> releases are about. I know that when a major release comes out that I
> should expect library changes that may break my code. If I don't want
> that pain, then I don't upgrade.
>
>> On the side, not leaving valued community members behind is important.
>>
>> I think the pmc / committers just need to make a decision which will
>> impact one group or the other.
>>
>> Chuck
>>
>>
>> Grant Ingersoll wrote on 06/13/2006 03:35 AM:
>>
>>
>>> Well, we have our first Java 1.5 patch... Now that we have had a week
>>> or two to digest the comments, do we want to reopen the discussion?
>>>
>>> Chuck Williams (JIRA) wrote:
>>>
>>>
>>>> [ http://issues.apache.org/jira/browse/LUCENE-600?page=all ]
>>>>
>>>> Chuck Williams updated LUCENE-600:
>>>> ----------------------------------
>>>>
>>>> Attachment: ParallelWriter.patch
>>>>
>>>> Patch to create and integrate ParallelWriter, Writable and
>>>> TestParallelWriter -- also modifies build to use java 1.5.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> ParallelWriter companion to ParallelReader
>>>>> ------------------------------------------
>>>>>
>>>>> Key: LUCENE-600
>>>>> URL: http://issues.apache.org/jira/browse/LUCENE-600
>>>>> Project: Lucene - Java
>>>>> Type: Improvement
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>> Components: Index
>>>>> Versions: 2.1
>>>>> Reporter: Chuck Williams
>>>>> Attachments: ParallelWriter.patch
>>>>>
>>>>> A new class ParallelWriter is provided that serves as a companion to
>>>>> ParallelReader. ParallelWriter meets all of the doc-id
>>>>> synchronization requirements of ParallelReader, subject to:
>>>>> 1. ParallelWriter.addDocument() is synchronized, which might
>>>>> have an adverse effect on performance. The writes to the
>>>>> sub-indexes are, however, done in parallel.
>>>>> 2. The application must ensure that the ParallelReader is never
>>>>> reopened inside ParallelWriter.addDocument(), else it might find the
>>>>> sub-indexes out of sync.
>>>>> 3. The application must deal with recovery from
>>>>> ParallelWriter.addDocument() exceptions. Recovery must restore the
>>>>> synchronization of doc-ids, e.g. by deleting any trailing
>>>>> document(s) in one sub-index that were not successfully added to all
>>>>> sub-indexes, and then optimizing all sub-indexes.
>>>>> A new interface, Writable, is provided to abstract IndexWriter and
>>>>> ParallelWriter. This is in the same spirit as the existing
>>>>> Searchable and Fieldable classes.
>>>>> This implementation uses java 1.5. The patch applies against
>>>>> today's svn head. All tests pass, including the new
>>>>> TestParallelWriter.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>>
>
>
--
Grant Ingersoll
Sr. Software Engineer
Center for Natural Language Processing
Syracuse University
School of Information Studies
335 Hinds Hall
Syracuse, NY 13244
http://www.cnlp.org
Voice: 315-443-5484
Fax: 315-443-6886
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Java 1.5 was [jira] Updated: (LUCENE-600) ParallelWriter companion
to ParallelReader
Posted by Grant Ingersoll <gs...@syr.edu>.
+1
Do you want to post it on the user list? It might also be good to put
it up on the main website.
Otis Gospodnetic wrote:
> Grant: how to poll users? How about this: http://www.quimble.com/poll/view/2156 ? If you think that's ok, we can send that to java-user tomorrow and see. Hey, how about some bets? I'll put a $10 for a beer on 1.5.
>
>
Wow, $10 for a beer? That must be some pretty good beer. Either that
or you live in New York City and that is a cheap beer! Anyway, I am
betting it is 1.5 as well. Maybe we can get together at ApacheCon or
something for one...
> Otis
>
> ----- Original Message ----
> From: Grant Ingersoll <gs...@syr.edu>
> To: java-dev@lucene.apache.org
> Sent: Tuesday, June 13, 2006 5:01:30 PM
> Subject: Re: Java 1.5 was [jira] Updated: (LUCENE-600) ParallelWriter companion to ParallelReader
>
>
>
>> In addition to performance, productivity and functionality benefits, my
>> main argument for 1.5 is that it is used by the vast majority of lucene
>> community members.
>>
>
> I am not so sure about this. Perhaps we should take a poll on the user
> list? Not even sure how that would be managed or counted, but...
>
>
>> Everything I write is in 1.5 and I don't have time
>> to backport. I have a significant body of code from which to extract
>> and contribute patches that others would likely find useful. How many
>> others are in a similar position?
>>
>>
> I definitely would prefer to make future contributions in 1.5 (even the
> patch we just contributed (issue 545) could have been better given 1.5,
> but it is fine with 1.4 as well). I tend to think if people don't want
> the new functionality or if it breaks their app. then they need not
> upgrade, or they can contribute patches against the branches for prior
> releases and we can support that as needed. To me, this is what major
> releases are about. I know that when a major release comes out that I
> should expect library changes that may break my code. If I don't want
> that pain, then I don't upgrade.
>
>> On the side, not leaving valued community members behind is important.
>>
>> I think the pmc / committers just need to make a decision which will
>> impact one group or the other.
>>
>> Chuck
>>
>>
>> Grant Ingersoll wrote on 06/13/2006 03:35 AM:
>>
>>
>>> Well, we have our first Java 1.5 patch... Now that we have had a week
>>> or two to digest the comments, do we want to reopen the discussion?
>>>
>>> Chuck Williams (JIRA) wrote:
>>>
>>>
>>>> [ http://issues.apache.org/jira/browse/LUCENE-600?page=all ]
>>>>
>>>> Chuck Williams updated LUCENE-600:
>>>> ----------------------------------
>>>>
>>>> Attachment: ParallelWriter.patch
>>>>
>>>> Patch to create and integrate ParallelWriter, Writable and
>>>> TestParallelWriter -- also modifies build to use java 1.5.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> ParallelWriter companion to ParallelReader
>>>>> ------------------------------------------
>>>>>
>>>>> Key: LUCENE-600
>>>>> URL: http://issues.apache.org/jira/browse/LUCENE-600
>>>>> Project: Lucene - Java
>>>>> Type: Improvement
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>> Components: Index
>>>>> Versions: 2.1
>>>>> Reporter: Chuck Williams
>>>>> Attachments: ParallelWriter.patch
>>>>>
>>>>> A new class ParallelWriter is provided that serves as a companion to
>>>>> ParallelReader. ParallelWriter meets all of the doc-id
>>>>> synchronization requirements of ParallelReader, subject to:
>>>>> 1. ParallelWriter.addDocument() is synchronized, which might
>>>>> have an adverse effect on performance. The writes to the
>>>>> sub-indexes are, however, done in parallel.
>>>>> 2. The application must ensure that the ParallelReader is never
>>>>> reopened inside ParallelWriter.addDocument(), else it might find the
>>>>> sub-indexes out of sync.
>>>>> 3. The application must deal with recovery from
>>>>> ParallelWriter.addDocument() exceptions. Recovery must restore the
>>>>> synchronization of doc-ids, e.g. by deleting any trailing
>>>>> document(s) in one sub-index that were not successfully added to all
>>>>> sub-indexes, and then optimizing all sub-indexes.
>>>>> A new interface, Writable, is provided to abstract IndexWriter and
>>>>> ParallelWriter. This is in the same spirit as the existing
>>>>> Searchable and Fieldable classes.
>>>>> This implementation uses java 1.5. The patch applies against
>>>>> today's svn head. All tests pass, including the new
>>>>> TestParallelWriter.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>>
>>
>
>
--
Grant Ingersoll
Sr. Software Engineer
Center for Natural Language Processing
Syracuse University
School of Information Studies
335 Hinds Hall
Syracuse, NY 13244
http://www.cnlp.org
Voice: 315-443-5484
Fax: 315-443-6886
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Java 1.5 was [jira] Updated: (LUCENE-600) ParallelWriter companion to ParallelReader
Posted by Otis Gospodnetic <ot...@yahoo.com>.
I agree and completely understand Chuck. I'm waiting for my employer to sign and fax the CCLA for some search benchmarking code I wrote, and it uses Java 1.5 stuff. It would only be a contrib piece, not core, so it's less of a problem, but...
Grant: how to poll users? How about this: http://www.quimble.com/poll/view/2156 ? If you think that's ok, we can send that to java-user tomorrow and see. Hey, how about some bets? I'll put a $10 for a beer on 1.5.
Otis
----- Original Message ----
From: Grant Ingersoll <gs...@syr.edu>
To: java-dev@lucene.apache.org
Sent: Tuesday, June 13, 2006 5:01:30 PM
Subject: Re: Java 1.5 was [jira] Updated: (LUCENE-600) ParallelWriter companion to ParallelReader
> In addition to performance, productivity and functionality benefits, my
> main argument for 1.5 is that it is used by the vast majority of lucene
> community members.
I am not so sure about this. Perhaps we should take a poll on the user
list? Not even sure how that would be managed or counted, but...
> Everything I write is in 1.5 and I don't have time
> to backport. I have a significant body of code from which to extract
> and contribute patches that others would likely find useful. How many
> others are in a similar position?
>
I definitely would prefer to make future contributions in 1.5 (even the
patch we just contributed (issue 545) could have been better given 1.5,
but it is fine with 1.4 as well). I tend to think if people don't want
the new functionality or if it breaks their app. then they need not
upgrade, or they can contribute patches against the branches for prior
releases and we can support that as needed. To me, this is what major
releases are about. I know that when a major release comes out that I
should expect library changes that may break my code. If I don't want
that pain, then I don't upgrade.
> On the side, not leaving valued community members behind is important.
>
> I think the pmc / committers just need to make a decision which will
> impact one group or the other.
>
> Chuck
>
>
> Grant Ingersoll wrote on 06/13/2006 03:35 AM:
>
>> Well, we have our first Java 1.5 patch... Now that we have had a week
>> or two to digest the comments, do we want to reopen the discussion?
>>
>> Chuck Williams (JIRA) wrote:
>>
>>> [ http://issues.apache.org/jira/browse/LUCENE-600?page=all ]
>>>
>>> Chuck Williams updated LUCENE-600:
>>> ----------------------------------
>>>
>>> Attachment: ParallelWriter.patch
>>>
>>> Patch to create and integrate ParallelWriter, Writable and
>>> TestParallelWriter -- also modifies build to use java 1.5.
>>>
>>>
>>>
>>>
>>>> ParallelWriter companion to ParallelReader
>>>> ------------------------------------------
>>>>
>>>> Key: LUCENE-600
>>>> URL: http://issues.apache.org/jira/browse/LUCENE-600
>>>> Project: Lucene - Java
>>>> Type: Improvement
>>>>
>>>>
>>>
>>>
>>>> Components: Index
>>>> Versions: 2.1
>>>> Reporter: Chuck Williams
>>>> Attachments: ParallelWriter.patch
>>>>
>>>> A new class ParallelWriter is provided that serves as a companion to
>>>> ParallelReader. ParallelWriter meets all of the doc-id
>>>> synchronization requirements of ParallelReader, subject to:
>>>> 1. ParallelWriter.addDocument() is synchronized, which might
>>>> have an adverse effect on performance. The writes to the
>>>> sub-indexes are, however, done in parallel.
>>>> 2. The application must ensure that the ParallelReader is never
>>>> reopened inside ParallelWriter.addDocument(), else it might find the
>>>> sub-indexes out of sync.
>>>> 3. The application must deal with recovery from
>>>> ParallelWriter.addDocument() exceptions. Recovery must restore the
>>>> synchronization of doc-ids, e.g. by deleting any trailing
>>>> document(s) in one sub-index that were not successfully added to all
>>>> sub-indexes, and then optimizing all sub-indexes.
>>>> A new interface, Writable, is provided to abstract IndexWriter and
>>>> ParallelWriter. This is in the same spirit as the existing
>>>> Searchable and Fieldable classes.
>>>> This implementation uses java 1.5. The patch applies against
>>>> today's svn head. All tests pass, including the new
>>>> TestParallelWriter.
>>>>
>>>>
>>>
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
--
Grant Ingersoll
Sr. Software Engineer
Center for Natural Language Processing
Syracuse University
School of Information Studies
335 Hinds Hall
Syracuse, NY 13244
http://www.cnlp.org
Voice: 315-443-5484
Fax: 315-443-6886
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Java 1.5 was [jira] Updated: (LUCENE-600) ParallelWriter companion
to ParallelReader
Posted by Grant Ingersoll <gs...@syr.edu>.
> In addition to performance, productivity and functionality benefits, my
> main argument for 1.5 is that it is used by the vast majority of lucene
> community members.
I am not so sure about this. Perhaps we should take a poll on the user
list? Not even sure how that would be managed or counted, but...
> Everything I write is in 1.5 and I don't have time
> to backport. I have a significant body of code from which to extract
> and contribute patches that others would likely find useful. How many
> others are in a similar position?
>
I definitely would prefer to make future contributions in 1.5 (even the
patch we just contributed (issue 545) could have been better given 1.5,
but it is fine with 1.4 as well). I tend to think if people don't want
the new functionality or if it breaks their app. then they need not
upgrade, or they can contribute patches against the branches for prior
releases and we can support that as needed. To me, this is what major
releases are about. I know that when a major release comes out that I
should expect library changes that may break my code. If I don't want
that pain, then I don't upgrade.
> On the side, not leaving valued community members behind is important.
>
> I think the pmc / committers just need to make a decision which will
> impact one group or the other.
>
> Chuck
>
>
> Grant Ingersoll wrote on 06/13/2006 03:35 AM:
>
>> Well, we have our first Java 1.5 patch... Now that we have had a week
>> or two to digest the comments, do we want to reopen the discussion?
>>
>> Chuck Williams (JIRA) wrote:
>>
>>> [ http://issues.apache.org/jira/browse/LUCENE-600?page=all ]
>>>
>>> Chuck Williams updated LUCENE-600:
>>> ----------------------------------
>>>
>>> Attachment: ParallelWriter.patch
>>>
>>> Patch to create and integrate ParallelWriter, Writable and
>>> TestParallelWriter -- also modifies build to use java 1.5.
>>>
>>>
>>>
>>>
>>>> ParallelWriter companion to ParallelReader
>>>> ------------------------------------------
>>>>
>>>> Key: LUCENE-600
>>>> URL: http://issues.apache.org/jira/browse/LUCENE-600
>>>> Project: Lucene - Java
>>>> Type: Improvement
>>>>
>>>>
>>>
>>>
>>>> Components: Index
>>>> Versions: 2.1
>>>> Reporter: Chuck Williams
>>>> Attachments: ParallelWriter.patch
>>>>
>>>> A new class ParallelWriter is provided that serves as a companion to
>>>> ParallelReader. ParallelWriter meets all of the doc-id
>>>> synchronization requirements of ParallelReader, subject to:
>>>> 1. ParallelWriter.addDocument() is synchronized, which might
>>>> have an adverse effect on performance. The writes to the
>>>> sub-indexes are, however, done in parallel.
>>>> 2. The application must ensure that the ParallelReader is never
>>>> reopened inside ParallelWriter.addDocument(), else it might find the
>>>> sub-indexes out of sync.
>>>> 3. The application must deal with recovery from
>>>> ParallelWriter.addDocument() exceptions. Recovery must restore the
>>>> synchronization of doc-ids, e.g. by deleting any trailing
>>>> document(s) in one sub-index that were not successfully added to all
>>>> sub-indexes, and then optimizing all sub-indexes.
>>>> A new interface, Writable, is provided to abstract IndexWriter and
>>>> ParallelWriter. This is in the same spirit as the existing
>>>> Searchable and Fieldable classes.
>>>> This implementation uses java 1.5. The patch applies against
>>>> today's svn head. All tests pass, including the new
>>>> TestParallelWriter.
>>>>
>>>>
>>>
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
--
Grant Ingersoll
Sr. Software Engineer
Center for Natural Language Processing
Syracuse University
School of Information Studies
335 Hinds Hall
Syracuse, NY 13244
http://www.cnlp.org
Voice: 315-443-5484
Fax: 315-443-6886
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Java 1.5 was [jira] Updated: (LUCENE-600) ParallelWriter companion
to ParallelReader
Posted by Chuck Williams <ch...@manawiz.com>.
I think the last discussion ended with the main counter-argument being
lack of support by gjc. Current top of GJC News:
> *June 6, 2006* RMS approved the plan to use the Eclipse compiler as
> the new gcj front end. Work is being done on the |gcj-eclipse| branch;
> it can already build libgcj. This project will allow us to ship a 1.5
> compiler in the relatively near future. The old |gcjx| branch and
> project is now dead.
In addition to performance, productivity and functionality benefits, my
main argument for 1.5 is that it is used by the vast majority of lucene
community members. Everything I write is in 1.5 and I don't have time
to backport. I have a significant body of code from which to extract
and contribute patches that others would likely find useful. How many
others are in a similar position?
On the side, not leaving valued community members behind is important.
I think the pmc / committers just need to make a decision which will
impact one group or the other.
Chuck
Grant Ingersoll wrote on 06/13/2006 03:35 AM:
> Well, we have our first Java 1.5 patch... Now that we have had a week
> or two to digest the comments, do we want to reopen the discussion?
>
> Chuck Williams (JIRA) wrote:
>> [ http://issues.apache.org/jira/browse/LUCENE-600?page=all ]
>>
>> Chuck Williams updated LUCENE-600:
>> ----------------------------------
>>
>> Attachment: ParallelWriter.patch
>>
>> Patch to create and integrate ParallelWriter, Writable and
>> TestParallelWriter -- also modifies build to use java 1.5.
>>
>>
>>
>>> ParallelWriter companion to ParallelReader
>>> ------------------------------------------
>>>
>>> Key: LUCENE-600
>>> URL: http://issues.apache.org/jira/browse/LUCENE-600
>>> Project: Lucene - Java
>>> Type: Improvement
>>>
>>
>>
>>> Components: Index
>>> Versions: 2.1
>>> Reporter: Chuck Williams
>>> Attachments: ParallelWriter.patch
>>>
>>> A new class ParallelWriter is provided that serves as a companion to
>>> ParallelReader. ParallelWriter meets all of the doc-id
>>> synchronization requirements of ParallelReader, subject to:
>>> 1. ParallelWriter.addDocument() is synchronized, which might
>>> have an adverse effect on performance. The writes to the
>>> sub-indexes are, however, done in parallel.
>>> 2. The application must ensure that the ParallelReader is never
>>> reopened inside ParallelWriter.addDocument(), else it might find the
>>> sub-indexes out of sync.
>>> 3. The application must deal with recovery from
>>> ParallelWriter.addDocument() exceptions. Recovery must restore the
>>> synchronization of doc-ids, e.g. by deleting any trailing
>>> document(s) in one sub-index that were not successfully added to all
>>> sub-indexes, and then optimizing all sub-indexes.
>>> A new interface, Writable, is provided to abstract IndexWriter and
>>> ParallelWriter. This is in the same spirit as the existing
>>> Searchable and Fieldable classes.
>>> This implementation uses java 1.5. The patch applies against
>>> today's svn head. All tests pass, including the new
>>> TestParallelWriter.
>>>
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Java 1.5 was [jira] Updated: (LUCENE-600) ParallelWriter companion
to ParallelReader
Posted by Grant Ingersoll <gs...@syr.edu>.
Well, we have our first Java 1.5 patch... Now that we have had a week
or two to digest the comments, do we want to reopen the discussion?
Chuck Williams (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/LUCENE-600?page=all ]
>
> Chuck Williams updated LUCENE-600:
> ----------------------------------
>
> Attachment: ParallelWriter.patch
>
> Patch to create and integrate ParallelWriter, Writable and TestParallelWriter -- also modifies build to use java 1.5.
>
>
>
>> ParallelWriter companion to ParallelReader
>> ------------------------------------------
>>
>> Key: LUCENE-600
>> URL: http://issues.apache.org/jira/browse/LUCENE-600
>> Project: Lucene - Java
>> Type: Improvement
>>
>
>
>> Components: Index
>> Versions: 2.1
>> Reporter: Chuck Williams
>> Attachments: ParallelWriter.patch
>>
>> A new class ParallelWriter is provided that serves as a companion to ParallelReader. ParallelWriter meets all of the doc-id synchronization requirements of ParallelReader, subject to:
>> 1. ParallelWriter.addDocument() is synchronized, which might have an adverse effect on performance. The writes to the sub-indexes are, however, done in parallel.
>> 2. The application must ensure that the ParallelReader is never reopened inside ParallelWriter.addDocument(), else it might find the sub-indexes out of sync.
>> 3. The application must deal with recovery from ParallelWriter.addDocument() exceptions. Recovery must restore the synchronization of doc-ids, e.g. by deleting any trailing document(s) in one sub-index that were not successfully added to all sub-indexes, and then optimizing all sub-indexes.
>> A new interface, Writable, is provided to abstract IndexWriter and ParallelWriter. This is in the same spirit as the existing Searchable and Fieldable classes.
>> This implementation uses java 1.5. The patch applies against today's svn head. All tests pass, including the new TestParallelWriter.
>>
>
>
--
Grant Ingersoll
Sr. Software Engineer
Center for Natural Language Processing
Syracuse University
School of Information Studies
335 Hinds Hall
Syracuse, NY 13244
http://www.cnlp.org
Voice: 315-443-5484
Fax: 315-443-6886
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org