You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Laurence Rowe <l...@lrowe.co.uk> on 2008/12/03 21:16:28 UTC
Ordering updates
Hi,
Our CMS is distributed over a cluster and I was wandering how I can
ensure that index records of newer versions of documents are never
overwritten by older ones. Amazon AWS uses a timestamp on requests to
ensure 'eventual consistency' of operations. Is there a way to supply
a transaction ID with an update so an update is conditional on the
supplied transaction id being greater than the existing indexed
transaction id?
Laurence
Re: Ordering updates
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Fri, Dec 5, 2008 at 5:40 AM, Laurence Rowe <la...@lrowe.co.uk> wrote:
> 2008/12/4 Shalin Shekhar Mangar <sh...@gmail.com>:
>
>
> I think we have a slight misunderstanding here. Because there are many
> CMS processes it is possible that the same document will be updated
> concurrently (from different web requests). In this case two updates
> are sent (one by each process). The problem arises when the two update
> requests are processed in a different order to the original database
> transactions.
Ok, I think I understand your problem now. If multiple processes send update
requests, they will overwrite each other which is not what you want.
> I guess the only way to achieve consistency is to stage my indexed
> data in a database table and trigger a DataImportHandler to perform
> delta imports after each transaction.
I agree. You need to use a transactional mechanism to ensure consistency,
then you should use a database. Periodically, you can index this particular
table into Solr. However, if you have multi-valued fields, you may run into
problems.
One more thing that you can think about, depending on your use-case, is
whether a small amount of stale data is OK? Do you really need things
consistent and upto date all the time in Solr? I also know of cases where
people have removed frequently changing fields from Solr and fetched them
from DB at the time of page render. Ofcourse, that doesn't work when you
need to sort by that frequently changind field.
> >> From what I can tell this conditional indexing feature is not
> >> supported by Solr. Might it be supported by Lucene but not exposed by
> >> Solr?
> >>
> >
> > No this is not supported by either of Lucene/Solr.
>
>
> This is a pity, eventual consistency is a nice model.
>
> Regards,
>
> Laurence
>
--
Regards,
Shalin Shekhar Mangar.
Re: Ordering updates
Posted by Laurence Rowe <la...@lrowe.co.uk>.
2008/12/4 Shalin Shekhar Mangar <sh...@gmail.com>:
> If each CMS process has a consistent view of the data and it wishes to
> update Solr with that data, where is the question of inconsistency here?
Because it is difficult to guarantee the order of processing across a
distributed system while minimising the time shared locks are held.
There is an interesting discussion on this here:
http://www.allthingsdistributed.com/2007/12/eventually_consistent.html
>
>> But I cannot see how to ensure that the reindexing
>> operations (that result in a delete and add for the document) are
>> processed in the order R1 then R2, rather than R2 then R1. In the
>> second case the index record is now inconsistent with the content
>> object in the database.
>
>
> When you need to update a document in Solr, you need to send the complete
> document and it will automatically do the replace. They will be visible to
> searchers when you call commit on Solr. From your CMS's perspective, it is a
> single operation. I hope I am understanding your problem correctly.
I think we have a slight misunderstanding here. Because there are many
CMS processes it is possible that the same document will be updated
concurrently (from different web requests). In this case two updates
are sent (one by each process). The problem arises when the two update
requests are processed in a different order to the original database
transactions.
I guess the only way to achieve consistency is to stage my indexed
data in a database table and trigger a DataImportHandler to perform
delta imports after each transaction.
>> From what I can tell this conditional indexing feature is not
>> supported by Solr. Might it be supported by Lucene but not exposed by
>> Solr?
>>
>
> No this is not supported by either of Lucene/Solr.
This is a pity, eventual consistency is a nice model.
Regards,
Laurence
Re: Ordering updates
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Fri, Dec 5, 2008 at 2:42 AM, Laurence Rowe <la...@lrowe.co.uk> wrote:
>
> We currently have a single Solr server, with a single index. There are
> a number of CMS processes distributed over a number of servers, with
> each CMS process sending an update to the Solr index when changes are
> made to a content object.
>
> My concern is that a scenario is possible where a content object is
> changed and reindexed concurrently by two CMS processes. The database
> ensures consistency within the CMS, these transactions get comitted as
> T1 and T2.
If each CMS process has a consistent view of the data and it wishes to
update Solr with that data, where is the question of inconsistency here?
> But I cannot see how to ensure that the reindexing
> operations (that result in a delete and add for the document) are
> processed in the order R1 then R2, rather than R2 then R1. In the
> second case the index record is now inconsistent with the content
> object in the database.
When you need to update a document in Solr, you need to send the complete
document and it will automatically do the replace. They will be visible to
searchers when you call commit on Solr. From your CMS's perspective, it is a
single operation. I hope I am understanding your problem correctly.
I would like to supply a transaction id with the reindex request, and
> configure Solr such that a reindex operation is processed if and only
> if the supplied transaction id is greater than the currently indexed
> transaction id.
>
> Otherwise the only way I can see to guarantee consistency is 1) have
> index operations processed by a single writer, or 2) commit the index
> operation between database prepare and commit statements.
>
> The first is not desirable as we introduce a single point of failure
> (in addition to the single Solr server) and delay updating the index.
> The second is not desirable because it reduces the throughput of the
> database, and with a distributed Solr setup would not solve the
> problem.
>
> From what I can tell this conditional indexing feature is not
> supported by Solr. Might it be supported by Lucene but not exposed by
> Solr?
>
No this is not supported by either of Lucene/Solr.
--
Regards,
Shalin Shekhar Mangar.
Re: Ordering updates
Posted by Laurence Rowe <la...@lrowe.co.uk>.
Hi,
We currently have a single Solr server, with a single index. There are
a number of CMS processes distributed over a number of servers, with
each CMS process sending an update to the Solr index when changes are
made to a content object.
My concern is that a scenario is possible where a content object is
changed and reindexed concurrently by two CMS processes. The database
ensures consistency within the CMS, these transactions get comitted as
T1 and T2. But I cannot see how to ensure that the reindexing
operations (that result in a delete and add for the document) are
processed in the order R1 then R2, rather than R2 then R1. In the
second case the index record is now inconsistent with the content
object in the database.
I would like to supply a transaction id with the reindex request, and
configure Solr such that a reindex operation is processed if and only
if the supplied transaction id is greater than the currently indexed
transaction id.
Otherwise the only way I can see to guarantee consistency is 1) have
index operations processed by a single writer, or 2) commit the index
operation between database prepare and commit statements.
The first is not desirable as we introduce a single point of failure
(in addition to the single Solr server) and delay updating the index.
The second is not desirable because it reduces the throughput of the
database, and with a distributed Solr setup would not solve the
problem.
>From what I can tell this conditional indexing feature is not
supported by Solr. Might it be supported by Lucene but not exposed by
Solr?
Thanks,
Laurence
2008/12/4 Shalin Shekhar Mangar <sh...@gmail.com>:
> It is not clear how you are using Solr i.e. distributed vs single index.
>
> Summarily, Solr does not update documents. It overwrites the old document
> with the new one if an old document with the same uniqueKey exists in the
> index.
>
> Does that answer your question?
>
> On Thu, Dec 4, 2008 at 1:46 AM, Laurence Rowe <l...@lrowe.co.uk> wrote:
>
>> Hi,
>>
>> Our CMS is distributed over a cluster and I was wandering how I can
>> ensure that index records of newer versions of documents are never
>> overwritten by older ones. Amazon AWS uses a timestamp on requests to
>> ensure 'eventual consistency' of operations. Is there a way to supply
>> a transaction ID with an update so an update is conditional on the
>> supplied transaction id being greater than the existing indexed
>> transaction id?
>>
>> Laurence
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: Ordering updates
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
It is not clear how you are using Solr i.e. distributed vs single index.
Summarily, Solr does not update documents. It overwrites the old document
with the new one if an old document with the same uniqueKey exists in the
index.
Does that answer your question?
On Thu, Dec 4, 2008 at 1:46 AM, Laurence Rowe <l...@lrowe.co.uk> wrote:
> Hi,
>
> Our CMS is distributed over a cluster and I was wandering how I can
> ensure that index records of newer versions of documents are never
> overwritten by older ones. Amazon AWS uses a timestamp on requests to
> ensure 'eventual consistency' of operations. Is there a way to supply
> a transaction ID with an update so an update is conditional on the
> supplied transaction id being greater than the existing indexed
> transaction id?
>
> Laurence
>
--
Regards,
Shalin Shekhar Mangar.