You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Jason Rutherglen <ja...@gmail.com> on 2008/08/21 00:08:45 UTC
IndexReaderWriter
Hello all,
Michael McCandless and Yonik Seeley and I were discussing the
IndexReaderWriter which would be a combined IndexReader and IndexWriter
class. The name of the class can change. One key feature that can be added
is realtime search. Realtime search would enable Lucene to have database
like functionality whereby each update is immediately searchable as opposed
to today where each update is searchable after a commit. The Ocean code
implements realtime search and could provide the basis for this
functionality. I would like to get to an agreement on the scope and
functionality of the project and gather input.
Thanks!
Jason Rutherglen
Re: IndexReaderWriter
Posted by Andrew Zhang <zh...@gmail.com>.
On Thu, Aug 21, 2008 at 6:08 AM, Jason Rutherglen <
jason.rutherglen@gmail.com> wrote:
> Hello all,
>
> Michael McCandless and Yonik Seeley and I were discussing the
> IndexReaderWriter which would be a combined IndexReader and IndexWriter
> class. The name of the class can change. One key feature that can be added
> is realtime search. Realtime search would enable Lucene to have database
> like functionality whereby each update is immediately searchable as opposed
> to today where each update is searchable after a commit.
>
I like the idea very much!
> The Ocean code implements realtime search and could provide the basis for
> this functionality. I would like to get to an agreement on the scope and
> functionality of the project and gather input.
>
Is there any "isolation" concept? Will the change reflect to other
IndexReader before commit?
>
>
> Thanks!
>
> Jason Rutherglen
>
--
Best regards,
Andrew Zhang
db4o - database for Android: www.db4o.com
http://zhanghuangzhu.blogspot.com/
Re: IndexReaderWriter
Posted by Jason Rutherglen <ja...@gmail.com>.
Hi Grant,
The patch is basically independent of the Lucene trunk. I will finish up
some stuff on the wiki though. http://*wiki*.apache.org/*lucene*
-java/OceanRealtimeSearch<http://wiki.apache.org/lucene-java/OceanRealtimeSearch>
Jason
On Thu, Aug 21, 2008 at 10:33 AM, Grant Ingersoll <gs...@apache.org>wrote:
>
> On Aug 21, 2008, at 10:07 AM, Jason Rutherglen wrote:
>
> Is there someone who can help me with the committing to contrib? Karl
> Wettin perhaps? It would be good to work with someone on the committing so
> that all the concepts are flushed out and documented.
>
>
> Any committer can, you just need to keep pestering us, we all basically
> volunteer and assign as we feel comfortable with based on our commitments.
> :-) One thing at this point, I guess to do, is make sure the patch is up to
> date w/ trunk. Basically, anything you can do to make it as simple as
> possible for us to review would be appreciated, i.e. good tests that pass,
> documentation, tutorial on use, etc.
>
> I'll see if I can try out the patch soon.
>
>
>
> On Thu, Aug 21, 2008 at 9:53 AM, Grant Ingersoll <gs...@apache.org>wrote:
>
>>
>> On Aug 21, 2008, at 5:42 AM, Michael McCandless wrote:
>>
>>
>>> I agree: getting real-time search working natively in Lucene is an
>>> important goal.
>>>
>>
>> +1. It's gotta be one of the most popular questions I get asked in
>> training/consulting jobs.
>>
>>
>>>
>>> I think as an immediate step we should get Ocean committed in contrib?
>>> This way users can easily try it out, report back, etc.
>>>
>>
>> +1
>>
>>
>
Re: IndexReaderWriter
Posted by Grant Ingersoll <gs...@apache.org>.
On Aug 21, 2008, at 10:07 AM, Jason Rutherglen wrote:
> Is there someone who can help me with the committing to contrib?
> Karl Wettin perhaps? It would be good to work with someone on the
> committing so that all the concepts are flushed out and documented.
Any committer can, you just need to keep pestering us, we all
basically volunteer and assign as we feel comfortable with based on
our commitments. :-) One thing at this point, I guess to do, is make
sure the patch is up to date w/ trunk. Basically, anything you can do
to make it as simple as possible for us to review would be
appreciated, i.e. good tests that pass, documentation, tutorial on
use, etc.
I'll see if I can try out the patch soon.
>
>
> On Thu, Aug 21, 2008 at 9:53 AM, Grant Ingersoll
> <gs...@apache.org> wrote:
>
> On Aug 21, 2008, at 5:42 AM, Michael McCandless wrote:
>
>
> I agree: getting real-time search working natively in Lucene is an
> important goal.
>
> +1. It's gotta be one of the most popular questions I get asked in
> training/consulting jobs.
>
>
>
>
> I think as an immediate step we should get Ocean committed in contrib?
> This way users can easily try it out, report back, etc.
>
> +1
>
Re: IndexReaderWriter
Posted by Karl Wettin <ka...@gmail.com>.
21 aug 2008 kl. 16.07 skrev Jason Rutherglen:
> Is there someone who can help me with the committing to contrib?
> Karl Wettin perhaps? It would be good to work with someone on the
> committing so that all the concepts are flushed out and documented.
As everybody else I'm rather busy with all these other projects, but
I've been keeping track of what you have been up to and I'll review
and play around with your patch ASAP.
karl
>
>
> On Thu, Aug 21, 2008 at 9:53 AM, Grant Ingersoll
> <gs...@apache.org> wrote:
>
> On Aug 21, 2008, at 5:42 AM, Michael McCandless wrote:
>
>
> I agree: getting real-time search working natively in Lucene is an
> important goal.
>
> +1. It's gotta be one of the most popular questions I get asked in
> training/consulting jobs.
>
>
>
>
> I think as an immediate step we should get Ocean committed in contrib?
> This way users can easily try it out, report back, etc.
>
> +1
>
>
>
>
> Ocean is nice in that it operates above Lucene's core (using
> InstantiatedIndex/Reader to handle the "live" additions). Besides
> LUCENE-1314 (forcing IndexReader.reopen to return a clone even when no
> changes have been committed to the index), are there any other pending
> core changes that Ocean needs?
>
> Mike
>
> Jason Rutherglen wrote:
>
> Hello all,
>
> Michael McCandless and Yonik Seeley and I were discussing the
> IndexReaderWriter which would be a combined IndexReader and
> IndexWriter class. The name of the class can change. One key
> feature that can be added is realtime search. Realtime search would
> enable Lucene to have database like functionality whereby each
> update is immediately searchable as opposed to today where each
> update is searchable after a commit. The Ocean code implements
> realtime search and could provide the basis for this functionality.
> I would like to get to an agreement on the scope and functionality
> of the project and gather input.
>
> Thanks!
>
> Jason Rutherglen
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: IndexReaderWriter
Posted by Jason Rutherglen <ja...@gmail.com>.
Is there someone who can help me with the committing to contrib? Karl
Wettin perhaps? It would be good to work with someone on the committing so
that all the concepts are flushed out and documented.
On Thu, Aug 21, 2008 at 9:53 AM, Grant Ingersoll <gs...@apache.org>wrote:
>
> On Aug 21, 2008, at 5:42 AM, Michael McCandless wrote:
>
>
>> I agree: getting real-time search working natively in Lucene is an
>> important goal.
>>
>
> +1. It's gotta be one of the most popular questions I get asked in
> training/consulting jobs.
>
>
>>
>> I think as an immediate step we should get Ocean committed in contrib?
>> This way users can easily try it out, report back, etc.
>>
>
> +1
>
>
>
>>
>> Ocean is nice in that it operates above Lucene's core (using
>> InstantiatedIndex/Reader to handle the "live" additions). Besides
>> LUCENE-1314 (forcing IndexReader.reopen to return a clone even when no
>> changes have been committed to the index), are there any other pending
>> core changes that Ocean needs?
>>
>> Mike
>>
>> Jason Rutherglen wrote:
>>
>> Hello all,
>>>
>>> Michael McCandless and Yonik Seeley and I were discussing the
>>> IndexReaderWriter which would be a combined IndexReader and IndexWriter
>>> class. The name of the class can change. One key feature that can be added
>>> is realtime search. Realtime search would enable Lucene to have database
>>> like functionality whereby each update is immediately searchable as opposed
>>> to today where each update is searchable after a commit. The Ocean code
>>> implements realtime search and could provide the basis for this
>>> functionality. I would like to get to an agreement on the scope and
>>> functionality of the project and gather input.
>>>
>>> Thanks!
>>>
>>> Jason Rutherglen
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: IndexReaderWriter
Posted by Grant Ingersoll <gs...@apache.org>.
On Aug 21, 2008, at 5:42 AM, Michael McCandless wrote:
>
> I agree: getting real-time search working natively in Lucene is an
> important goal.
+1. It's gotta be one of the most popular questions I get asked in
training/consulting jobs.
>
>
> I think as an immediate step we should get Ocean committed in contrib?
> This way users can easily try it out, report back, etc.
+1
>
>
> Ocean is nice in that it operates above Lucene's core (using
> InstantiatedIndex/Reader to handle the "live" additions). Besides
> LUCENE-1314 (forcing IndexReader.reopen to return a clone even when no
> changes have been committed to the index), are there any other pending
> core changes that Ocean needs?
>
> Mike
>
> Jason Rutherglen wrote:
>
>> Hello all,
>>
>> Michael McCandless and Yonik Seeley and I were discussing the
>> IndexReaderWriter which would be a combined IndexReader and
>> IndexWriter class. The name of the class can change. One key
>> feature that can be added is realtime search. Realtime search
>> would enable Lucene to have database like functionality whereby
>> each update is immediately searchable as opposed to today where
>> each update is searchable after a commit. The Ocean code
>> implements realtime search and could provide the basis for this
>> functionality. I would like to get to an agreement on the scope
>> and functionality of the project and gather input.
>>
>> Thanks!
>>
>> Jason Rutherglen
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: IndexReaderWriter
Posted by Jason Rutherglen <ja...@gmail.com>.
Michael,
The realtime search code is more or less ready. It probably needs some more
test cases related to multithreading. I could use some help in coming up
with test cases as admittedly I struggle with them.
In terms of the native Lucene version, I think it may be useful to offer
optimistic concurrency which would make the API slightly different than
IndexWriter's add, delete, update document methods. The reason optimistic
concurrency is needed is updating documents can become problematic without
it in realtime.
The API might look something like:
addDocument()
updateDocument(String id, long version, Document document)
deleteDocument(String id)
On Thu, Aug 21, 2008 at 5:42 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> I agree: getting real-time search working natively in Lucene is an
> important goal.
>
> I think as an immediate step we should get Ocean committed in contrib?
> This way users can easily try it out, report back, etc.
>
> Ocean is nice in that it operates above Lucene's core (using
> InstantiatedIndex/Reader to handle the "live" additions). Besides
> LUCENE-1314 (forcing IndexReader.reopen to return a clone even when no
> changes have been committed to the index), are there any other pending
> core changes that Ocean needs?
>
> Mike
>
>
> Jason Rutherglen wrote:
>
> Hello all,
>>
>> Michael McCandless and Yonik Seeley and I were discussing the
>> IndexReaderWriter which would be a combined IndexReader and IndexWriter
>> class. The name of the class can change. One key feature that can be added
>> is realtime search. Realtime search would enable Lucene to have database
>> like functionality whereby each update is immediately searchable as opposed
>> to today where each update is searchable after a commit. The Ocean code
>> implements realtime search and could provide the basis for this
>> functionality. I would like to get to an agreement on the scope and
>> functionality of the project and gather input.
>>
>> Thanks!
>>
>> Jason Rutherglen
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re: IndexReaderWriter
Posted by Michael McCandless <lu...@mikemccandless.com>.
I agree: getting real-time search working natively in Lucene is an
important goal.
I think as an immediate step we should get Ocean committed in contrib?
This way users can easily try it out, report back, etc.
Ocean is nice in that it operates above Lucene's core (using
InstantiatedIndex/Reader to handle the "live" additions). Besides
LUCENE-1314 (forcing IndexReader.reopen to return a clone even when no
changes have been committed to the index), are there any other pending
core changes that Ocean needs?
Mike
Jason Rutherglen wrote:
> Hello all,
>
> Michael McCandless and Yonik Seeley and I were discussing the
> IndexReaderWriter which would be a combined IndexReader and
> IndexWriter class. The name of the class can change. One key
> feature that can be added is realtime search. Realtime search would
> enable Lucene to have database like functionality whereby each
> update is immediately searchable as opposed to today where each
> update is searchable after a commit. The Ocean code implements
> realtime search and could provide the basis for this functionality.
> I would like to get to an agreement on the scope and functionality
> of the project and gather input.
>
> Thanks!
>
> Jason Rutherglen
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org