You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Christian Rodriguez <cr...@gmail.com> on 2004/10/15 23:13:45 UTC
Atomicity in Lucene operations
Hello guys,
I need additions and deletions of documents to the index to be ATOMIC
(they either happen to completion or not at all).
On top of this, I need updates (which I currently implement with a
deletion of the document followed by an addition) to be ATOMIC and
DURABLE (once I return from the "update" function its because the
operation happened to completion and stays in the index).
Notice that I dont really need all the ACID properties for all the operations.
I have tried to solve the problem by using the Lucene + BDB package
written by Andi Vajda and using transactions, but the BDB database
gets corrupted if I insert random System.exit() to simulate a crash of
the application before aborting or commiting transactions.
So I have two questions:
1. Has anyone been able to use the Lucene + BDB WITH transactions and
simulate random crashes at different points in the process of addding
items and found it to be robust (specially, have you been able to
always recover after a crash, with uncommited txns rolled back and
commited ones present in the DB)?
2. Can anyone suggest other solutions (beside using BDB) that may
work? For example: are any of these operations already atomic in
Lucene (using an FSDirectory)?
Thanks for any help you can give me!
Xtian
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Atomicity in Lucene operations
Posted by Nader Henein <ns...@bayt.net>.
As soon as I've cleaned up the code, I'll publish it, it needs a little
more documentation as well.
Nader
Roy Shan wrote:
>Maybe you can contribute it to sandbox?
>
>
>On Mon, 18 Oct 2004 08:31:30 -0700 (PDT), Yonik Seeley
><ys...@yahoo.com> wrote:
>
>
>>Hi Nader,
>>I would greatly appreciate it if you could CC me on
>>the docs or the code.
>>
>>Thanks!
>>Yonik
>>
>>
>>--- Nader Henein <ns...@bayt.net> wrote:
>>
>>
>>
>>>It's pretty integrated into our system at this
>>>point, I'm working on
>>>Packaging it and cleaning up my documentation and
>>>then I'll make it
>>>available, I can give you the documents and if you
>>>still want the code
>>>I'll slap together a ruff copy for you and ship it
>>>across.
>>>
>>>
>>>Nader Henein
>>>
>>>Roy Shan wrote:
>>>
>>>
>>>
>>>>Hello, Nader:
>>>>
>>>>I am very interested in how you implement the
>>>>
>>>>
>>>atomicity. Could you
>>>
>>>
>>>>send me a copy of your code?
>>>>
>>>>Thanks in advance.
>>>>
>>>>Roy
>>>>
>>>>
>>
>>__________________________________
>>Do you Yahoo!?
>>Yahoo! Mail - Helps protect you from nasty viruses.
>>http://promotions.yahoo.com/new_mail
>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>>
>>
>>
>>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Atomicity in Lucene operations
Posted by Roy Shan <ro...@gmail.com>.
Maybe you can contribute it to sandbox?
On Mon, 18 Oct 2004 08:31:30 -0700 (PDT), Yonik Seeley
<ys...@yahoo.com> wrote:
> Hi Nader,
> I would greatly appreciate it if you could CC me on
> the docs or the code.
>
> Thanks!
> Yonik
>
>
> --- Nader Henein <ns...@bayt.net> wrote:
>
> > It's pretty integrated into our system at this
> > point, I'm working on
> > Packaging it and cleaning up my documentation and
> > then I'll make it
> > available, I can give you the documents and if you
> > still want the code
> > I'll slap together a ruff copy for you and ship it
> > across.
> >
> >
> > Nader Henein
> >
> > Roy Shan wrote:
> >
> > >Hello, Nader:
> > >
> > >I am very interested in how you implement the
> > atomicity. Could you
> > >send me a copy of your code?
> > >
> > >Thanks in advance.
> > >
> > >Roy
>
>
> __________________________________
> Do you Yahoo!?
> Yahoo! Mail - Helps protect you from nasty viruses.
> http://promotions.yahoo.com/new_mail
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
--
Roy
**May I open-source your mind?**
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Atomicity in Lucene operations
Posted by Yonik Seeley <ys...@yahoo.com>.
Hi Nader,
I would greatly appreciate it if you could CC me on
the docs or the code.
Thanks!
Yonik
--- Nader Henein <ns...@bayt.net> wrote:
> It's pretty integrated into our system at this
> point, I'm working on
> Packaging it and cleaning up my documentation and
> then I'll make it
> available, I can give you the documents and if you
> still want the code
> I'll slap together a ruff copy for you and ship it
> across.
>
>
> Nader Henein
>
> Roy Shan wrote:
>
> >Hello, Nader:
> >
> >I am very interested in how you implement the
> atomicity. Could you
> >send me a copy of your code?
> >
> >Thanks in advance.
> >
> >Roy
__________________________________
Do you Yahoo!?
Yahoo! Mail - Helps protect you from nasty viruses.
http://promotions.yahoo.com/new_mail
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Atomicity in Lucene operations
Posted by Nader Henein <ns...@bayt.net>.
It's pretty integrated into our system at this point, I'm working on
Packaging it and cleaning up my documentation and then I'll make it
available, I can give you the documents and if you still want the code
I'll slap together a ruff copy for you and ship it across.
Nader Henein
Roy Shan wrote:
>Hello, Nader:
>
>I am very interested in how you implement the atomicity. Could you
>send me a copy of your code?
>
>Thanks in advance.
>
>Roy
>
>
>
>On Sat, 16 Oct 2004 01:20:09 +0400, Nader Henein <ns...@bayt.net> wrote:
>
>
>>We use Lucene over 4 replicated indecies and we have to maintain
>>atomicity on deletion and updates with multiple fallback points. I'll
>>send you the right up, it's too big to CC the entire board.
>>
>>nader henein
>>
>>
>>
>>Christian Rodriguez wrote:
>>
>>
>>
>>>Hello guys,
>>>
>>>I need additions and deletions of documents to the index to be ATOMIC
>>>(they either happen to completion or not at all).
>>>
>>>On top of this, I need updates (which I currently implement with a
>>>deletion of the document followed by an addition) to be ATOMIC and
>>>DURABLE (once I return from the "update" function its because the
>>>operation happened to completion and stays in the index).
>>>
>>>Notice that I dont really need all the ACID properties for all the operations.
>>>
>>>I have tried to solve the problem by using the Lucene + BDB package
>>>written by Andi Vajda and using transactions, but the BDB database
>>>gets corrupted if I insert random System.exit() to simulate a crash of
>>>the application before aborting or commiting transactions.
>>>
>>>So I have two questions:
>>>1. Has anyone been able to use the Lucene + BDB WITH transactions and
>>>simulate random crashes at different points in the process of addding
>>>items and found it to be robust (specially, have you been able to
>>>always recover after a crash, with uncommited txns rolled back and
>>>commited ones present in the DB)?
>>>2. Can anyone suggest other solutions (beside using BDB) that may
>>>work? For example: are any of these operations already atomic in
>>>Lucene (using an FSDirectory)?
>>>
>>>Thanks for any help you can give me!
>>>Xtian
>>>
>>>---------------------------------------------------------------------
>>>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>>
>>
>>
>>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Atomicity in Lucene operations
Posted by Roy Shan <ro...@gmail.com>.
Hello, Nader:
I am very interested in how you implement the atomicity. Could you
send me a copy of your code?
Thanks in advance.
Roy
On Sat, 16 Oct 2004 01:20:09 +0400, Nader Henein <ns...@bayt.net> wrote:
> We use Lucene over 4 replicated indecies and we have to maintain
> atomicity on deletion and updates with multiple fallback points. I'll
> send you the right up, it's too big to CC the entire board.
>
> nader henein
>
>
>
> Christian Rodriguez wrote:
>
> >Hello guys,
> >
> >I need additions and deletions of documents to the index to be ATOMIC
> >(they either happen to completion or not at all).
> >
> >On top of this, I need updates (which I currently implement with a
> >deletion of the document followed by an addition) to be ATOMIC and
> >DURABLE (once I return from the "update" function its because the
> >operation happened to completion and stays in the index).
> >
> >Notice that I dont really need all the ACID properties for all the operations.
> >
> >I have tried to solve the problem by using the Lucene + BDB package
> >written by Andi Vajda and using transactions, but the BDB database
> >gets corrupted if I insert random System.exit() to simulate a crash of
> >the application before aborting or commiting transactions.
> >
> >So I have two questions:
> >1. Has anyone been able to use the Lucene + BDB WITH transactions and
> >simulate random crashes at different points in the process of addding
> >items and found it to be robust (specially, have you been able to
> >always recover after a crash, with uncommited txns rolled back and
> >commited ones present in the DB)?
> >2. Can anyone suggest other solutions (beside using BDB) that may
> >work? For example: are any of these operations already atomic in
> >Lucene (using an FSDirectory)?
> >
> >Thanks for any help you can give me!
> >Xtian
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> >For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> >
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
--
Roy
**May I open-source your mind?**
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Atomicity in Lucene operations
Posted by Nader Henein <ns...@bayt.net>.
We use Lucene over 4 replicated indecies and we have to maintain
atomicity on deletion and updates with multiple fallback points. I'll
send you the right up, it's too big to CC the entire board.
nader henein
Christian Rodriguez wrote:
>Hello guys,
>
>I need additions and deletions of documents to the index to be ATOMIC
>(they either happen to completion or not at all).
>
>On top of this, I need updates (which I currently implement with a
>deletion of the document followed by an addition) to be ATOMIC and
>DURABLE (once I return from the "update" function its because the
>operation happened to completion and stays in the index).
>
>Notice that I dont really need all the ACID properties for all the operations.
>
>I have tried to solve the problem by using the Lucene + BDB package
>written by Andi Vajda and using transactions, but the BDB database
>gets corrupted if I insert random System.exit() to simulate a crash of
>the application before aborting or commiting transactions.
>
>So I have two questions:
>1. Has anyone been able to use the Lucene + BDB WITH transactions and
>simulate random crashes at different points in the process of addding
>items and found it to be robust (specially, have you been able to
>always recover after a crash, with uncommited txns rolled back and
>commited ones present in the DB)?
>2. Can anyone suggest other solutions (beside using BDB) that may
>work? For example: are any of these operations already atomic in
>Lucene (using an FSDirectory)?
>
>Thanks for any help you can give me!
>Xtian
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org