You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by João Silva <ga...@gmail.com> on 2009/06/16 11:03:33 UTC
Index Concurrent access
Hi,
how can i access the index in a concurrently way,
so i can perform add/update/delete documents concurrently?
Cheers,
João
--
Cumprimentos,
João Carlos Galaio da Silva
Re: Index Concurrent access
Posted by João Silva <ga...@gmail.com>.
Thanks Manish for your fast answer.
I trying to implement a web tool for uploading documents,
for now i'm implementing basic operations, to upload the
and retrieve the users files, so it can read and/or
modify/delete them.
Imagine tha i have several users performing that operations,
is there any implementation(pool,threading, etc) in lucene
to perform that kindo of operations.
I already seen the Lucene-1026 that initially implemented some
kind of that functionality, but it was abandoned.
Is there any following implementations of that ticket?
Thanks,
João
On Tue, Jun 16, 2009 at 10:21 AM, Manish Joshi <mj...@zedo.com> wrote:
> Concurrently adding updating deleting may not be good idea.It may corrupt
> the index
> -Manish B. Joshi
>
> On Tue, Jun 16, 2009 at 2:33 PM, João Silva <ga...@gmail.com>
> wrote:
>
> > Hi,
> > how can i access the index in a concurrently way,
> > so i can perform add/update/delete documents concurrently?
> >
> > Cheers,
> > João
> >
> > --
> > Cumprimentos,
> > João Carlos Galaio da Silva
> >
>
>
>
> --
> Manish Joshi
> Adserving Team
> Zedo India
>
--
Cumprimentos,
João Carlos Galaio da Silva
Re: Index Concurrent access
Posted by Michael McCandless <lu...@mikemccandless.com>.
LUCENE-1313 is just an enhancement to near real-time search that won't
make it into 2.9 at this point (we are "gunning" to get 2.9 out the
door...).
Ie, near real-time search was already committed to trunk, under LUCENE-1516.
Mike
On Tue, Jun 16, 2009 at 7:08 AM, João Silva<ga...@gmail.com> wrote:
> Thanks mike, i will see that.
> The ticket for that functionallity is the Lucene-1313?
>
> Thanks,
> João
>
>
>
> On Tue, Jun 16, 2009 at 11:53 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> Have you tried the patch on LUCENE-1026? It's rather standalone from
>> Lucene's core -- it adds a convenience layer (for interleaving
>> reads/writes) on a single index.
>>
>> Or, code it up yourself. As of 2.9 (not yet released -- available on
>> trunk now), near real-time search makes this particularly simple.
>> You'd keep a writer open indefinitely, to do deletes/adds, then call
>> writer.getReader() to get a new IndexReader that sees all changes done
>> against that writer. You can then separately call writer.commit() to
>> make all changes permanent (written to stable storage) in the index,
>> as your app requires.
>>
>> Multiple users making changes should be simple, since IndexWriter is
>> thread safe..
>>
>> Mike
>>
>> On Tue, Jun 16, 2009 at 6:10 AM, João Silva<ga...@gmail.com> wrote:
>> > I mike, thanks.
>> >
>> > I rewrite my problem:
>> >
>> > I trying to implement a web tool for uploading documents,
>> > for now i'm implementing basic operations, to upload the
>> > and retrieve the users files, so it can read and/or
>> > modify/delete them.
>> > Imagine tha i have several users performing that operations,
>> > is there any implementation(pool,threading, etc) in lucene
>> > to perform that kindo of operations.
>> >
>> > I already seen the Lucene-1026 that initially implemented some
>> > kind of that functionality, but it was abandoned.
>> >
>> > Is there any following implementations of that ticket?
>> >
>> > Its a single jvm and tool will work in a webserver.
>> >
>> >
>> > Thanks,
>> > joão
>> >
>> >
>> >
>> >
>> >
>> > On Tue, Jun 16, 2009 at 11:07 AM, Michael McCandless <
>> > lucene@mikemccandless.com> wrote:
>> >
>> >> On Tue, Jun 16, 2009 at 6:03 AM, Simon
>> >> Willnauer<si...@googlemail.com> wrote:
>> >>
>> >> > Mike I guess you mean a single VM (JRE rather refers to a version or
>> >> > vendor) - Just wanna clarify.
>> >>
>> >> Right, I meant a "single java process", so I guess an instance of a JVM?
>> >>
>> >> Within that instance, many threads can be doing writing against an
>> index.
>> >>
>> >> Mike
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>
>> >>
>> >
>> >
>> > --
>> > Cumprimentos,
>> > João Carlos Galaio da Silva
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
> Cumprimentos,
> João Carlos Galaio da Silva
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Index Concurrent access
Posted by João Silva <ga...@gmail.com>.
Thanks mike, i will see that.
The ticket for that functionallity is the Lucene-1313?
Thanks,
João
On Tue, Jun 16, 2009 at 11:53 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> Have you tried the patch on LUCENE-1026? It's rather standalone from
> Lucene's core -- it adds a convenience layer (for interleaving
> reads/writes) on a single index.
>
> Or, code it up yourself. As of 2.9 (not yet released -- available on
> trunk now), near real-time search makes this particularly simple.
> You'd keep a writer open indefinitely, to do deletes/adds, then call
> writer.getReader() to get a new IndexReader that sees all changes done
> against that writer. You can then separately call writer.commit() to
> make all changes permanent (written to stable storage) in the index,
> as your app requires.
>
> Multiple users making changes should be simple, since IndexWriter is
> thread safe..
>
> Mike
>
> On Tue, Jun 16, 2009 at 6:10 AM, João Silva<ga...@gmail.com> wrote:
> > I mike, thanks.
> >
> > I rewrite my problem:
> >
> > I trying to implement a web tool for uploading documents,
> > for now i'm implementing basic operations, to upload the
> > and retrieve the users files, so it can read and/or
> > modify/delete them.
> > Imagine tha i have several users performing that operations,
> > is there any implementation(pool,threading, etc) in lucene
> > to perform that kindo of operations.
> >
> > I already seen the Lucene-1026 that initially implemented some
> > kind of that functionality, but it was abandoned.
> >
> > Is there any following implementations of that ticket?
> >
> > Its a single jvm and tool will work in a webserver.
> >
> >
> > Thanks,
> > joão
> >
> >
> >
> >
> >
> > On Tue, Jun 16, 2009 at 11:07 AM, Michael McCandless <
> > lucene@mikemccandless.com> wrote:
> >
> >> On Tue, Jun 16, 2009 at 6:03 AM, Simon
> >> Willnauer<si...@googlemail.com> wrote:
> >>
> >> > Mike I guess you mean a single VM (JRE rather refers to a version or
> >> > vendor) - Just wanna clarify.
> >>
> >> Right, I meant a "single java process", so I guess an instance of a JVM?
> >>
> >> Within that instance, many threads can be doing writing against an
> index.
> >>
> >> Mike
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >
> >
> > --
> > Cumprimentos,
> > João Carlos Galaio da Silva
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Cumprimentos,
João Carlos Galaio da Silva
Re: Index Concurrent access
Posted by Michael McCandless <lu...@mikemccandless.com>.
Have you tried the patch on LUCENE-1026? It's rather standalone from
Lucene's core -- it adds a convenience layer (for interleaving
reads/writes) on a single index.
Or, code it up yourself. As of 2.9 (not yet released -- available on
trunk now), near real-time search makes this particularly simple.
You'd keep a writer open indefinitely, to do deletes/adds, then call
writer.getReader() to get a new IndexReader that sees all changes done
against that writer. You can then separately call writer.commit() to
make all changes permanent (written to stable storage) in the index,
as your app requires.
Multiple users making changes should be simple, since IndexWriter is
thread safe..
Mike
On Tue, Jun 16, 2009 at 6:10 AM, João Silva<ga...@gmail.com> wrote:
> I mike, thanks.
>
> I rewrite my problem:
>
> I trying to implement a web tool for uploading documents,
> for now i'm implementing basic operations, to upload the
> and retrieve the users files, so it can read and/or
> modify/delete them.
> Imagine tha i have several users performing that operations,
> is there any implementation(pool,threading, etc) in lucene
> to perform that kindo of operations.
>
> I already seen the Lucene-1026 that initially implemented some
> kind of that functionality, but it was abandoned.
>
> Is there any following implementations of that ticket?
>
> Its a single jvm and tool will work in a webserver.
>
>
> Thanks,
> joão
>
>
>
>
>
> On Tue, Jun 16, 2009 at 11:07 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> On Tue, Jun 16, 2009 at 6:03 AM, Simon
>> Willnauer<si...@googlemail.com> wrote:
>>
>> > Mike I guess you mean a single VM (JRE rather refers to a version or
>> > vendor) - Just wanna clarify.
>>
>> Right, I meant a "single java process", so I guess an instance of a JVM?
>>
>> Within that instance, many threads can be doing writing against an index.
>>
>> Mike
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
> Cumprimentos,
> João Carlos Galaio da Silva
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Index Concurrent access
Posted by João Silva <ga...@gmail.com>.
I mike, thanks.
I rewrite my problem:
I trying to implement a web tool for uploading documents,
for now i'm implementing basic operations, to upload the
and retrieve the users files, so it can read and/or
modify/delete them.
Imagine tha i have several users performing that operations,
is there any implementation(pool,threading, etc) in lucene
to perform that kindo of operations.
I already seen the Lucene-1026 that initially implemented some
kind of that functionality, but it was abandoned.
Is there any following implementations of that ticket?
Its a single jvm and tool will work in a webserver.
Thanks,
joão
On Tue, Jun 16, 2009 at 11:07 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> On Tue, Jun 16, 2009 at 6:03 AM, Simon
> Willnauer<si...@googlemail.com> wrote:
>
> > Mike I guess you mean a single VM (JRE rather refers to a version or
> > vendor) - Just wanna clarify.
>
> Right, I meant a "single java process", so I guess an instance of a JVM?
>
> Within that instance, many threads can be doing writing against an index.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Cumprimentos,
João Carlos Galaio da Silva
Re: Index Concurrent access
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Tue, Jun 16, 2009 at 6:03 AM, Simon
Willnauer<si...@googlemail.com> wrote:
> Mike I guess you mean a single VM (JRE rather refers to a version or
> vendor) - Just wanna clarify.
Right, I meant a "single java process", so I guess an instance of a JVM?
Within that instance, many threads can be doing writing against an index.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Index Concurrent access
Posted by Simon Willnauer <si...@googlemail.com>.
On Tue, Jun 16, 2009 at 11:59 AM, Michael
McCandless<lu...@mikemccandless.com> wrote:
> Concurrency using multiple threads in a single JRE is perfectly fine
> and strongly encouraged, since modern hardware is very concurrent.
>
> But, concurrent access via different JREs is not supported by Lucene.
> Lucene's write lock will protect the index against such cases, but if
> you bypass the write lock then you will easily corrupt the index if
> two writers are opened at once.
>
> Mike
Mike I guess you mean a single VM (JRE rather refers to a version or
vendor) - Just wanna clarify.
simon
>
> On Tue, Jun 16, 2009 at 5:21 AM, Manish Joshi<mj...@zedo.com> wrote:
>> Concurrently adding updating deleting may not be good idea.It may corrupt
>> the index
>> -Manish B. Joshi
>>
>> On Tue, Jun 16, 2009 at 2:33 PM, João Silva <ga...@gmail.com> wrote:
>>
>>> Hi,
>>> how can i access the index in a concurrently way,
>>> so i can perform add/update/delete documents concurrently?
>>>
>>> Cheers,
>>> João
>>>
>>> --
>>> Cumprimentos,
>>> João Carlos Galaio da Silva
>>>
>>
>>
>>
>> --
>> Manish Joshi
>> Adserving Team
>> Zedo India
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Index Concurrent access
Posted by Michael McCandless <lu...@mikemccandless.com>.
Concurrency using multiple threads in a single JRE is perfectly fine
and strongly encouraged, since modern hardware is very concurrent.
But, concurrent access via different JREs is not supported by Lucene.
Lucene's write lock will protect the index against such cases, but if
you bypass the write lock then you will easily corrupt the index if
two writers are opened at once.
Mike
On Tue, Jun 16, 2009 at 5:21 AM, Manish Joshi<mj...@zedo.com> wrote:
> Concurrently adding updating deleting may not be good idea.It may corrupt
> the index
> -Manish B. Joshi
>
> On Tue, Jun 16, 2009 at 2:33 PM, João Silva <ga...@gmail.com> wrote:
>
>> Hi,
>> how can i access the index in a concurrently way,
>> so i can perform add/update/delete documents concurrently?
>>
>> Cheers,
>> João
>>
>> --
>> Cumprimentos,
>> João Carlos Galaio da Silva
>>
>
>
>
> --
> Manish Joshi
> Adserving Team
> Zedo India
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Index Concurrent access
Posted by Manish Joshi <mj...@zedo.com>.
Concurrently adding updating deleting may not be good idea.It may corrupt
the index
-Manish B. Joshi
On Tue, Jun 16, 2009 at 2:33 PM, João Silva <ga...@gmail.com> wrote:
> Hi,
> how can i access the index in a concurrently way,
> so i can perform add/update/delete documents concurrently?
>
> Cheers,
> João
>
> --
> Cumprimentos,
> João Carlos Galaio da Silva
>
--
Manish Joshi
Adserving Team
Zedo India