Does solr support two phase commit or any other distributed transaction protocol?

[Pablo Anzorena <an...@gmail.com>]

That's it.

Thanks.

Re: Does solr support two phase commit or any other distributed transaction protocol?

[Walter Underwood <wu...@wunderwood.org>]

Solr does not have transactions.

A batch is submitted, then processed. The command to process the batch is named “commit”, but it isn’t very much like a database commit.

Batch submissions are not isolated between clients. If three batches are being submitted at the same time, a commit command from one client will cause all pending documents to be processed, not just the documents from that client.

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/  (my blog)


> On Aug 9, 2016, at 6:55 AM, Pablo Anzorena <an...@gmail.com> wrote:
> 
> That's it.
> 
> Thanks.

Re: Does solr support two phase commit or any other distributed transaction protocol?

[Pablo Anzorena <an...@gmail.com>]

Thanks Shawn, I understood perfectly well. One important thing in my use
case is that I only have one entry point for indexing solr, so I won't have
any problems of multiple threads trying to update the index.

So what can I do if I have to index in solr and also in postgres and I need
to do it transactionally?

I imagine something like:
1)Open a distributed transaction in postgresql and "index" the data with
the global id transaction.
1.1)If some problem occurs, rollback postgres. End of transaction.
2)Index data in solr. If no problem occurs, commit in solr and then commit
in postgres. End of transaction.
2.1)If some problem occurs in solr, rollback solr and rollback postgres.
End of transaction.

2016-08-09 11:24 GMT-03:00 Shawn Heisey <ap...@elyograg.org>:

> On 8/9/2016 7:55 AM, Pablo Anzorena wrote:
> > That's it. Thanks.
>
> Solr doesn't support transactions in the way that most people with a
> database background imagine them.
>
> With a typical database server, all changes to the database that happen
> on a single DB connection can be committed or rolled back completely
> independently from updates that happen on other DB connections.
>
> Solr doesn't work this way.
>
> In a Lucene index (Solr is a Lucene program), a "transaction" is all
> updates made since the last commit with openSearcher=true.  This
> includes ALL updates made, regardless of where they came from.  So if
> you have a dozen different threads/processes making changes to your Solr
> index, then have something do a commit, all of the updates made by those
> 12 sources before the commit will be committed.  There is no concept of
> an individual transaction.
>
> Adding the DB transaction model would be a *major* development effort,
> and there's a good chance that adding it would destroy the blazing
> search performance that Solr and Lucene are known for.
>
> Thanks,
> Shawn
>
>

Re: Does solr support two phase commit or any other distributed transaction protocol?

[Shawn Heisey <ap...@elyograg.org>]

On 8/9/2016 7:55 AM, Pablo Anzorena wrote:
> That's it. Thanks.

Solr doesn't support transactions in the way that most people with a
database background imagine them.

With a typical database server, all changes to the database that happen
on a single DB connection can be committed or rolled back completely
independently from updates that happen on other DB connections.

Solr doesn't work this way.

In a Lucene index (Solr is a Lucene program), a "transaction" is all
updates made since the last commit with openSearcher=true.  This
includes ALL updates made, regardless of where they came from.  So if
you have a dozen different threads/processes making changes to your Solr
index, then have something do a commit, all of the updates made by those
12 sources before the commit will be committed.  There is no concept of
an individual transaction.

Adding the DB transaction model would be a *major* development effort,
and there's a good chance that adding it would destroy the blazing
search performance that Solr and Lucene are known for.

Thanks,
Shawn