You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Oscar Picasso <os...@yahoo.com> on 2005/02/14 19:02:23 UTC
Transactional Directories
Hi,
I am currently implementing a Directory backed by a Berkeley DB that I am
willing to release as an open source project.
Besides the internal implementation, it differs from the one in the sandbox in
that it is implemented with the Berkeley DB Java Edition.
Using the Java Edition allows an easier distribution as you just need to add a
single jar in your classpath and you have a fully functional Berkeley DB
embedded in your application without the hassle of installing the C Berkeley
DB.
While initially implemented with the Java Edition this Directory can easily be
ported to a Berkeley DB C edition or a Berkeley DB XML (for example to use
Berkeley DB XML + Lucene as the base for a document management system).
This implementation works fine and I am quite happy with its speed.
There is still an important problem I face and it has to do with how to deal
with some transactions. After all, the purpose of a Berkeley implementation, or
a JDBC one for that matter, is its ability to use transactions.
After looking at the Andy Varga code, it seems that the implementation in the
sandbox face the same problem (correct me if I am wrong). I have also learn
that the JDBC directory was not implemented with transactions in mind.
Here the problem.
If I do something like that:
-- case A --
<pseudo-code>
+begin transaction
new IndexWriter
create/update/delete objects in the database
index.addDocument (related to the objects)
indexWriter.close()
+commit
</pseudo-code>
Everything is fine. The operations are transactionally protected. You can even
do many writes/updates. As far as everything in enclosed by the pairs
begin-transaction/new-index-writer ... index-writer.close/commit everything is
properly undone is case of any operation fails insidde the transaction.
For batch insertions the whole batch is rolled back but at least your object
database is consistent with the index.
If you do mostly batch insertions and relatively few random individual
insertions. That's fine.
However with a relatively high number of random insertions, the cost of the
"new IndexWriter / index.close()" performed for each insertion is two high.
Unfortunately this it is a common case for some kind of applications and it is
where a transactional directory would the most useful.
In such a case you would like to do something like that:
-- case B --
<pseudo-code>
new IndexWriter
...
+begin transaction-1
create/update/delete objects in the database
index.addDocument (related to the objects)
+ commit
...
+begin transaction-2
create/update/delete objects in the database
index.addDocument (related to the objects)
+ commit
...
indexWriter.close()
</pseudo-code>
The benefits would be to protect individual insertions while avoiding the cost
of using each time a new IndexWriter.
It doesn't work however. Here is my understanding.
Suppose that in case B, transaction-1 fails and transaction-2 succeeds.
In that case the underlying database system rolls back all the writes done
during transaction-1 whether they were related to the objects stored in the
database or to the index (the writes done to the IndexOutput are also undone).
>From the database point of view consistency is maintained between the stored
object and the index.
The problem is that after transaction-1 Lucene still 'remembers' the segment(s)
it wrote during transaction-1. Later, Lucene might 'want' to perform some
operation based on these references (on merging the segments, I think) while
the underlying segment(s) files do not exist anymore. This is where an
Exception is thrown.
The solution would be to instruct Lucene to 'forget' or undo any reference to
the segments created during transaction-1 in case of rollback;
I have noticed that references to the segments are stored in a segmentInfos
map. I was thinking about removing the segmentsInfo map entries created during
transaction-1 in case of a rollback but I don't see if it's enough and/or
potentially dangerous.
I would really appreciate any comment about this idea and also about my
understanding of the Lucene indexing process.
If I/we could find a solution it would also benefit a JDBC Directory
implementation
Thanks.
Oscar
P.S.: If and when my implementation is fully functional, is there a place in
the Lucene project where I could release it? (Maybe the sandbox).
__________________________________
Do you Yahoo!?
The all-new My Yahoo! - What will yours do?
http://my.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Transactional Directories
Posted by Doug Cutting <cu...@apache.org>.
Oscar Picasso wrote:
> Hi,
>
> I am currently implementing a Directory backed by a Berkeley DB that I am
> willing to release as an open source project.
>
> Besides the internal implementation, it differs from the one in the sandbox in
> that it is implemented with the Berkeley DB Java Edition.
>
> Using the Java Edition allows an easier distribution as you just need to add a
> single jar in your classpath and you have a fully functional Berkeley DB
> embedded in your application without the hassle of installing the C Berkeley
> DB.
>
> While initially implemented with the Java Edition this Directory can easily be
> ported to a Berkeley DB C edition or a Berkeley DB XML (for example to use
> Berkeley DB XML + Lucene as the base for a document management system).
>
> This implementation works fine and I am quite happy with its speed.
>
> There is still an important problem I face and it has to do with how to deal
> with some transactions. After all, the purpose of a Berkeley implementation, or
> a JDBC one for that matter, is its ability to use transactions.
>
> After looking at the Andy Varga code, it seems that the implementation in the
> sandbox face the same problem (correct me if I am wrong). I have also learn
> that the JDBC directory was not implemented with transactions in mind.
>
> Here the problem.
>
> If I do something like that:
> -- case A --
> <pseudo-code>
> +begin transaction
> new IndexWriter
> create/update/delete objects in the database
> index.addDocument (related to the objects)
> indexWriter.close()
> +commit
> </pseudo-code>
>
> Everything is fine. The operations are transactionally protected. You can even
> do many writes/updates. As far as everything in enclosed by the pairs
> begin-transaction/new-index-writer ... index-writer.close/commit everything is
> properly undone is case of any operation fails insidde the transaction.
>
> For batch insertions the whole batch is rolled back but at least your object
> database is consistent with the index.
>
> If you do mostly batch insertions and relatively few random individual
> insertions. That's fine.
>
> However with a relatively high number of random insertions, the cost of the
> "new IndexWriter / index.close()" performed for each insertion is two high.
> Unfortunately this it is a common case for some kind of applications and it is
> where a transactional directory would the most useful.
>
> In such a case you would like to do something like that:
> -- case B --
> <pseudo-code>
> new IndexWriter
> ...
> +begin transaction-1
> create/update/delete objects in the database
> index.addDocument (related to the objects)
> + commit
> ...
> +begin transaction-2
> create/update/delete objects in the database
> index.addDocument (related to the objects)
> + commit
> ...
> indexWriter.close()
> </pseudo-code>
>
> The benefits would be to protect individual insertions while avoiding the cost
> of using each time a new IndexWriter.
>
> It doesn't work however. Here is my understanding.
>
> Suppose that in case B, transaction-1 fails and transaction-2 succeeds.
>
> In that case the underlying database system rolls back all the writes done
> during transaction-1 whether they were related to the objects stored in the
> database or to the index (the writes done to the IndexOutput are also undone).
> From the database point of view consistency is maintained between the stored
> object and the index.
>
> The problem is that after transaction-1 Lucene still 'remembers' the segment(s)
> it wrote during transaction-1. Later, Lucene might 'want' to perform some
> operation based on these references (on merging the segments, I think) while
> the underlying segment(s) files do not exist anymore. This is where an
> Exception is thrown.
>
> The solution would be to instruct Lucene to 'forget' or undo any reference to
> the segments created during transaction-1 in case of rollback;
>
> I have noticed that references to the segments are stored in a segmentInfos
> map. I was thinking about removing the segmentsInfo map entries created during
> transaction-1 in case of a rollback but I don't see if it's enough and/or
> potentially dangerous.
>
> I would really appreciate any comment about this idea and also about my
> understanding of the Lucene indexing process.
>
> If I/we could find a solution it would also benefit a JDBC Directory
> implementation
>
> Thanks.
>
> Oscar
>
> P.S.: If and when my implementation is fully functional, is there a place in
> the Lucene project where I could release it? (Maybe the sandbox).
>
>
>
>
> __________________________________
> Do you Yahoo!?
> The all-new My Yahoo! - What will yours do?
> http://my.yahoo.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Transactional Directories
Posted by Oscar Picasso <os...@yahoo.com>.
--- Doug Cutting <cu...@apache.org> wrote:
> [...]
> > However with a relatively high number of random insertions, the cost of the
> > "new IndexWriter / index.close()" performed for each insertion is two high.
>
> Did you measure that? How much slower was it? Did you perform any
> profiling?
I made 100 consecutive insertions and repeated 5 times for each test. Each
insertion was enclosed into a transaction.
a - the average time for 100 insertions when using the same IndexWriter for
each batch of 100 insertions is 1486 ms.
b - the average time for 100 insertions when using a new IndexWriter for each
individual insertion is 14135 ms.
Test a is around 10 times faster than b. Unfortunately b is when the
transactional behavior is not respected.
> Perhaps one could improve this by, e.g., disabling document
> index buffering, so that indexes are written directly to the final
> directory in this case, rather than first buffered in a RAMDirectory.
I don't understand here. As to guarantee that the transactions are safe I need,
until there is a better solution, to enclose each insertion into a pair new
IndexWriter.. indexWriter.close(), what could be the impact of disabling
document buffering.
Anyway, I tried setMaxBufferedDocs(1) with the configuration b. The results
didn't change significantly. I tried also setMaxBufferedDocs(0) but the
application didn't return. It seems that in this case Lucene enters into an
endless loop.
> So you've got multiple threads? Or are you proceeding in the face of
> exceptions? Otherwise I would expect that if transaction-1 fails then
> you'd avoid transaction-2, no?
In a real application it would be typically multiple threads. My 'pseudo-code'
was not very clear. It didn't mean that transaction-2 was after transaction-1
in the code, but that transaction-2 happens later in the execution of the
application.
Actually for my testing I did something like this:
for(int i = 0; i < 100; i++)
{
...
try
{
insert(myItem);
}
catch(Exception)
{
logger.error("Your item was not inserted");
}
}
..
void insert(Item item)
{
beginTransaction;
indexWriter.addDocument(item.toDocument());
insert item in database;
// Simulation of something wrong
if(Math.random() > 0.5)
{
throw Exception("Something wrong");
}
commit;
}
In a similar case if a transaction fails, it doesn't impact transactions that
occur later in the loop.
>
> Also, you'd want to add an flush() call after each addDocument(), since
> document additions are bufferred. But a flush() is just what
> IndexWriter.close() does, so then things would not be any faster than
> creating a new IndexWriter for each document.
I guess you meant IndexWriter.flushRammSegments()? I did that and as you said
it wasn't faster.
> The bottom line is that there are optimizations to be made when batching
> additions. Lucene's API is designed to encourage batching, so that
> these optimizations may be used. If you don't batch, things will be
> somewhat slower.
In this case it's 10 times slower.
So you think that creating a custom IndexWriter that would remove segmentInfos
entries (or maybe other entries) in case of transaction failure is a too
simplistic approach?
Oscar
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
Re: Transactional Directories
Posted by Doug Cutting <cu...@apache.org>.
[ Please ignore my previous message. I somehow hit "Send" before typing
anything! ]
Oscar Picasso wrote:
> However with a relatively high number of random insertions, the cost of the
> "new IndexWriter / index.close()" performed for each insertion is two high.
Did you measure that? How much slower was it? Did you perform any
profiling? Perhaps one could improve this by, e.g., disabling document
index buffering, so that indexes are written directly to the final
directory in this case, rather than first bufferred in a RAMDirectory.
> Unfortunately this it is a common case for some kind of applications and it is
> where a transactional directory would the most useful.
>
> In such a case you would like to do something like that:
> -- case B --
> <pseudo-code>
> new IndexWriter
> ...
> +begin transaction-1
> create/update/delete objects in the database
> index.addDocument (related to the objects)
> + commit
> ...
> +begin transaction-2
> create/update/delete objects in the database
> index.addDocument (related to the objects)
> + commit
> ...
> indexWriter.close()
> </pseudo-code>
>
> The benefits would be to protect individual insertions while avoiding the cost
> of using each time a new IndexWriter.
>
> It doesn't work however. Here is my understanding.
>
> Suppose that in case B, transaction-1 fails and transaction-2 succeeds.
So you've got multiple threads? Or are you proceeding in the face of
exceptions? Otherwise I would expect that if transaction-1 fails then
you'd avoid transaction-2, no?
Also, you'd want to add an flush() call after each addDocument(), since
document additions are bufferred. But a flush() is just what
IndexWriter.close() does, so then things would not be any faster than
creating a new IndexWriter for each document.
The bottom line is that there are optimizations to be made when batching
additions. Lucene's API is designed to encourage batching, so that
these optimizations may be used. If you don't batch, things will be
somewhat slower.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org