You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Amr Ali <am...@siliconexpert.com> on 2015/05/12 18:58:36 UTC
Transactional Behavior
Hello,
I have a business case in which I need to be able for the rollback. When I tried add/commit I was not able to prevent other threads that write to a given Solr core from committing everything. I also tried indexwriter but Solr did not get changes until we restart it.
--
Regards,
Amr Ali
City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
Ext: 278
Re: Transactional Behavior
Posted by Walter Underwood <wu...@wunderwood.org>.
Solr does not do transactions at all. The “commit” is to signal the end of a batch load and start the processing.
You can hack some things, but there is no way to roll back a delete or a update.
If you need this, you need to use a database, not a search engine. I recommend MarkLogic. It is a full-featured database that includes search features.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
On May 14, 2015, at 6:12 AM, Emir Arnautovic <em...@sematext.com> wrote:
> Hi Amr,
> As far as I am aware, SOLR does not support transaction isolation. If you really need it, you will have to isolate it yourself. Easiest way is to sacrifice concurrent updates.
> Other way of achieving it would be to to index in separate indices (maybe in memory) concurrently and than merge it with your main index in isolation. There are probably better solutions but this is from the top of my head.
>
> Thanks,
> Emir
>
> On 14.05.2015 14:21, Amr Ali wrote:
>> Hello guys,
>>
>> Any expert advice this issue?
>>
>>
>> --
>> Regards,
>> Amr Ali
>>
>> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
>> Ext: 278
>>
>>
>>
>> -----Original Message-----
>> From: Amr Ali
>> Sent: Wednesday, May 13, 2015 10:58 AM
>> To: solr-user@lucene.apache.org
>> Subject: RE: Transactional Behavior
>>
>> Hello Emir,
>>
>> But this is not a transaction because if some of the bulk I need to add is committed; they will be searchable. In a transaction I need to insert a bulk of data (all bulk data will be searchable once) or roll it back according to some business scenarios.
>>
>> --
>> Regards,
>> Amr Ali
>>
>> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
>> Ext: 278
>>
>>
>>
>> -----Original Message-----
>> From: Emir Arnautovic [mailto:emir.arnautovic@sematext.com]
>> Sent: Tuesday, May 12, 2015 10:46 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Transactional Behavior
>>
>> Hi Amr,
>> One option is to include transaction id in your documents and do delete in case of failed transaction. It is not cheap option - additional field if you don't have something to use to identify transaction. Assuming rollback will not happen to often deleting is not that big issue.
>>
>> Thanks,
>> Emir
>>
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/
>>
>>
>>
>> On 12.05.2015 22:37, Amr Ali wrote:
>>> Please check this
>>>
>>> https://lucene.apache.org/solr/4_1_0/solr-solrj/org/apache/solr/client
>>> /solrj/SolrServer.html#rollback() "Note that this is not a true
>>> rollback as in databases. Content you have previously added may have been committed due to autoCommit, buffer full, other client performing a commit etc."
>>>
>>> It is not a real rollback if you have two threads T1 and T2 that are adding. If T1 is adding 500 and T2 is adding 3 then T2 will commit its 3 document PLUS the documents added by T1 (because T2 will finish add/commit before T2 due to the documents number). Solr transactions are "server side" only.
>>>
>>>
>>> --
>>> Regards,
>>> Amr Ali
>>>
>>> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
>>> Ext: 278
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Jack Krupansky [mailto:jack.krupansky@gmail.com]
>>> Sent: Tuesday, May 12, 2015 10:24 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: Transactional Behavior
>>>
>>> Solr does have a <rollback/> command, but it is an expert feature and not so clear how it works in SolrCloud.
>>>
>>> See:
>>> https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+I
>>> ndex+Handlers
>>> and
>>> https://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22
>>>
>>>
>>> -- Jack Krupansky
>>>
>>> On Tue, May 12, 2015 at 12:58 PM, Amr Ali <am...@siliconexpert.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I have a business case in which I need to be able for the rollback.
>>>> When I tried add/commit I was not able to prevent other threads that
>>>> write to a given Solr core from committing everything. I also tried
>>>> indexwriter but Solr did not get changes until we restart it.
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Amr Ali
>>>>
>>>> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
>>>> Ext: 278
>>>>
>>>>
>>>>
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/
>>
>
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
Re: Transactional Behavior
Posted by Emir Arnautovic <em...@sematext.com>.
Hi Amr,
As far as I am aware, SOLR does not support transaction isolation. If
you really need it, you will have to isolate it yourself. Easiest way is
to sacrifice concurrent updates.
Other way of achieving it would be to to index in separate indices
(maybe in memory) concurrently and than merge it with your main index in
isolation. There are probably better solutions but this is from the top
of my head.
Thanks,
Emir
On 14.05.2015 14:21, Amr Ali wrote:
> Hello guys,
>
> Any expert advice this issue?
>
>
> --
> Regards,
> Amr Ali
>
> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
> Ext: 278
>
>
>
> -----Original Message-----
> From: Amr Ali
> Sent: Wednesday, May 13, 2015 10:58 AM
> To: solr-user@lucene.apache.org
> Subject: RE: Transactional Behavior
>
> Hello Emir,
>
> But this is not a transaction because if some of the bulk I need to add is committed; they will be searchable. In a transaction I need to insert a bulk of data (all bulk data will be searchable once) or roll it back according to some business scenarios.
>
> --
> Regards,
> Amr Ali
>
> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
> Ext: 278
>
>
>
> -----Original Message-----
> From: Emir Arnautovic [mailto:emir.arnautovic@sematext.com]
> Sent: Tuesday, May 12, 2015 10:46 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Transactional Behavior
>
> Hi Amr,
> One option is to include transaction id in your documents and do delete in case of failed transaction. It is not cheap option - additional field if you don't have something to use to identify transaction. Assuming rollback will not happen to often deleting is not that big issue.
>
> Thanks,
> Emir
>
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/
>
>
>
> On 12.05.2015 22:37, Amr Ali wrote:
>> Please check this
>>
>> https://lucene.apache.org/solr/4_1_0/solr-solrj/org/apache/solr/client
>> /solrj/SolrServer.html#rollback() "Note that this is not a true
>> rollback as in databases. Content you have previously added may have been committed due to autoCommit, buffer full, other client performing a commit etc."
>>
>> It is not a real rollback if you have two threads T1 and T2 that are adding. If T1 is adding 500 and T2 is adding 3 then T2 will commit its 3 document PLUS the documents added by T1 (because T2 will finish add/commit before T2 due to the documents number). Solr transactions are "server side" only.
>>
>>
>> --
>> Regards,
>> Amr Ali
>>
>> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
>> Ext: 278
>>
>>
>>
>> -----Original Message-----
>> From: Jack Krupansky [mailto:jack.krupansky@gmail.com]
>> Sent: Tuesday, May 12, 2015 10:24 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Transactional Behavior
>>
>> Solr does have a <rollback/> command, but it is an expert feature and not so clear how it works in SolrCloud.
>>
>> See:
>> https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+I
>> ndex+Handlers
>> and
>> https://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22
>>
>>
>> -- Jack Krupansky
>>
>> On Tue, May 12, 2015 at 12:58 PM, Amr Ali <am...@siliconexpert.com> wrote:
>>
>>> Hello,
>>>
>>> I have a business case in which I need to be able for the rollback.
>>> When I tried add/commit I was not able to prevent other threads that
>>> write to a given Solr core from committing everything. I also tried
>>> indexwriter but Solr did not get changes until we restart it.
>>>
>>>
>>> --
>>> Regards,
>>> Amr Ali
>>>
>>> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
>>> Ext: 278
>>>
>>>
>>>
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/
>
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
RE: Transactional Behavior
Posted by Amr Ali <am...@siliconexpert.com>.
Hello guys,
Any expert advice this issue?
--
Regards,
Amr Ali
City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
Ext: 278
-----Original Message-----
From: Amr Ali
Sent: Wednesday, May 13, 2015 10:58 AM
To: solr-user@lucene.apache.org
Subject: RE: Transactional Behavior
Hello Emir,
But this is not a transaction because if some of the bulk I need to add is committed; they will be searchable. In a transaction I need to insert a bulk of data (all bulk data will be searchable once) or roll it back according to some business scenarios.
--
Regards,
Amr Ali
City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
Ext: 278
-----Original Message-----
From: Emir Arnautovic [mailto:emir.arnautovic@sematext.com]
Sent: Tuesday, May 12, 2015 10:46 PM
To: solr-user@lucene.apache.org
Subject: Re: Transactional Behavior
Hi Amr,
One option is to include transaction id in your documents and do delete in case of failed transaction. It is not cheap option - additional field if you don't have something to use to identify transaction. Assuming rollback will not happen to often deleting is not that big issue.
Thanks,
Emir
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/
On 12.05.2015 22:37, Amr Ali wrote:
> Please check this
>
> https://lucene.apache.org/solr/4_1_0/solr-solrj/org/apache/solr/client
> /solrj/SolrServer.html#rollback() "Note that this is not a true
> rollback as in databases. Content you have previously added may have been committed due to autoCommit, buffer full, other client performing a commit etc."
>
> It is not a real rollback if you have two threads T1 and T2 that are adding. If T1 is adding 500 and T2 is adding 3 then T2 will commit its 3 document PLUS the documents added by T1 (because T2 will finish add/commit before T2 due to the documents number). Solr transactions are "server side" only.
>
>
> --
> Regards,
> Amr Ali
>
> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
> Ext: 278
>
>
>
> -----Original Message-----
> From: Jack Krupansky [mailto:jack.krupansky@gmail.com]
> Sent: Tuesday, May 12, 2015 10:24 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Transactional Behavior
>
> Solr does have a <rollback/> command, but it is an expert feature and not so clear how it works in SolrCloud.
>
> See:
> https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+I
> ndex+Handlers
> and
> https://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22
>
>
> -- Jack Krupansky
>
> On Tue, May 12, 2015 at 12:58 PM, Amr Ali <am...@siliconexpert.com> wrote:
>
>> Hello,
>>
>> I have a business case in which I need to be able for the rollback.
>> When I tried add/commit I was not able to prevent other threads that
>> write to a given Solr core from committing everything. I also tried
>> indexwriter but Solr did not get changes until we restart it.
>>
>>
>> --
>> Regards,
>> Amr Ali
>>
>> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
>> Ext: 278
>>
>>
>>
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/
RE: Transactional Behavior
Posted by Amr Ali <am...@siliconexpert.com>.
Hello Emir,
But this is not a transaction because if some of the bulk I need to add is committed; they will be searchable. In a transaction I need to insert a bulk of data (all bulk data will be searchable once) or roll it back according to some business scenarios.
--
Regards,
Amr Ali
City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
Ext: 278
-----Original Message-----
From: Emir Arnautovic [mailto:emir.arnautovic@sematext.com]
Sent: Tuesday, May 12, 2015 10:46 PM
To: solr-user@lucene.apache.org
Subject: Re: Transactional Behavior
Hi Amr,
One option is to include transaction id in your documents and do delete in case of failed transaction. It is not cheap option - additional field if you don't have something to use to identify transaction. Assuming rollback will not happen to often deleting is not that big issue.
Thanks,
Emir
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/
On 12.05.2015 22:37, Amr Ali wrote:
> Please check this
>
> https://lucene.apache.org/solr/4_1_0/solr-solrj/org/apache/solr/client/solrj/SolrServer.html#rollback()
> "Note that this is not a true rollback as in databases. Content you have previously added may have been committed due to autoCommit, buffer full, other client performing a commit etc."
>
> It is not a real rollback if you have two threads T1 and T2 that are adding. If T1 is adding 500 and T2 is adding 3 then T2 will commit its 3 document PLUS the documents added by T1 (because T2 will finish add/commit before T2 due to the documents number). Solr transactions are "server side" only.
>
>
> --
> Regards,
> Amr Ali
>
> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
> Ext: 278
>
>
>
> -----Original Message-----
> From: Jack Krupansky [mailto:jack.krupansky@gmail.com]
> Sent: Tuesday, May 12, 2015 10:24 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Transactional Behavior
>
> Solr does have a <rollback/> command, but it is an expert feature and not so clear how it works in SolrCloud.
>
> See:
> https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers
> and
> https://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22
>
>
> -- Jack Krupansky
>
> On Tue, May 12, 2015 at 12:58 PM, Amr Ali <am...@siliconexpert.com> wrote:
>
>> Hello,
>>
>> I have a business case in which I need to be able for the rollback.
>> When I tried add/commit I was not able to prevent other threads that
>> write to a given Solr core from committing everything. I also tried
>> indexwriter but Solr did not get changes until we restart it.
>>
>>
>> --
>> Regards,
>> Amr Ali
>>
>> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
>> Ext: 278
>>
>>
>>
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
Re: Transactional Behavior
Posted by Emir Arnautovic <em...@sematext.com>.
Hi Amr,
One option is to include transaction id in your documents and do delete
in case of failed transaction. It is not cheap option - additional field
if you don't have something to use to identify transaction. Assuming
rollback will not happen to often deleting is not that big issue.
Thanks,
Emir
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On 12.05.2015 22:37, Amr Ali wrote:
> Please check this
>
> https://lucene.apache.org/solr/4_1_0/solr-solrj/org/apache/solr/client/solrj/SolrServer.html#rollback()
> "Note that this is not a true rollback as in databases. Content you have previously added may have been committed due to autoCommit, buffer full, other client performing a commit etc."
>
> It is not a real rollback if you have two threads T1 and T2 that are adding. If T1 is adding 500 and T2 is adding 3 then T2 will commit its 3 document PLUS the documents added by T1 (because T2 will finish add/commit before T2 due to the documents number). Solr transactions are "server side" only.
>
>
> --
> Regards,
> Amr Ali
>
> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
> Ext: 278
>
>
>
> -----Original Message-----
> From: Jack Krupansky [mailto:jack.krupansky@gmail.com]
> Sent: Tuesday, May 12, 2015 10:24 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Transactional Behavior
>
> Solr does have a <rollback/> command, but it is an expert feature and not so clear how it works in SolrCloud.
>
> See:
> https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers
> and
> https://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22
>
>
> -- Jack Krupansky
>
> On Tue, May 12, 2015 at 12:58 PM, Amr Ali <am...@siliconexpert.com> wrote:
>
>> Hello,
>>
>> I have a business case in which I need to be able for the rollback.
>> When I tried add/commit I was not able to prevent other threads that
>> write to a given Solr core from committing everything. I also tried
>> indexwriter but Solr did not get changes until we restart it.
>>
>>
>> --
>> Regards,
>> Amr Ali
>>
>> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
>> Ext: 278
>>
>>
>>
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
RE: Transactional Behavior
Posted by Amr Ali <am...@siliconexpert.com>.
Please check this
https://lucene.apache.org/solr/4_1_0/solr-solrj/org/apache/solr/client/solrj/SolrServer.html#rollback()
"Note that this is not a true rollback as in databases. Content you have previously added may have been committed due to autoCommit, buffer full, other client performing a commit etc."
It is not a real rollback if you have two threads T1 and T2 that are adding. If T1 is adding 500 and T2 is adding 3 then T2 will commit its 3 document PLUS the documents added by T1 (because T2 will finish add/commit before T2 due to the documents number). Solr transactions are "server side" only.
--
Regards,
Amr Ali
City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
Ext: 278
-----Original Message-----
From: Jack Krupansky [mailto:jack.krupansky@gmail.com]
Sent: Tuesday, May 12, 2015 10:24 PM
To: solr-user@lucene.apache.org
Subject: Re: Transactional Behavior
Solr does have a <rollback/> command, but it is an expert feature and not so clear how it works in SolrCloud.
See:
https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers
and
https://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22
-- Jack Krupansky
On Tue, May 12, 2015 at 12:58 PM, Amr Ali <am...@siliconexpert.com> wrote:
> Hello,
>
> I have a business case in which I need to be able for the rollback.
> When I tried add/commit I was not able to prevent other threads that
> write to a given Solr core from committing everything. I also tried
> indexwriter but Solr did not get changes until we restart it.
>
>
> --
> Regards,
> Amr Ali
>
> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
> Ext: 278
>
>
>
Re: Transactional Behavior
Posted by Jack Krupansky <ja...@gmail.com>.
Solr does have a <rollback/> command, but it is an expert feature and not
so clear how it works in SolrCloud.
See:
https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Index+Handlers
and
https://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22
-- Jack Krupansky
On Tue, May 12, 2015 at 12:58 PM, Amr Ali <am...@siliconexpert.com> wrote:
> Hello,
>
> I have a business case in which I need to be able for the rollback. When I
> tried add/commit I was not able to prevent other threads that write to a
> given Solr core from committing everything. I also tried indexwriter but
> Solr did not get changes until we restart it.
>
>
> --
> Regards,
> Amr Ali
>
> City stars capital 8 - 3rd floor, Nasr city, Cairo, Egypt
> Ext: 278
>
>
>
Re: Transactional Behavior
Posted by Shawn Heisey <ap...@elyograg.org>.
On 5/12/2015 10:58 AM, Amr Ali wrote:
> I have a business case in which I need to be able for the rollback. When I tried add/commit I was not able to prevent other threads that write to a given Solr core from committing everything. I also tried indexwriter but Solr did not get changes until we restart it.
This is how Lucene is designed. The bulk of Solr's functionality is
provided directly by Lucene, including the way that commits work.
Adding transaction support with individual rollback capability would be
a VERY fundamental and low-level change to Lucene. I bet you would find
that major pieces of Lucene functionality are dependent on the current
non-transactional design for commits, which means that changing it would
probably require changes in every major subsystem found in the code, and
that general performance might be very adversely affected. Verifying
correctness and fixing problems could take weeks or months.
I don't think the benefit is worth the pain, especially when there's a
simple workaround -- don't overlap changes that need to be treated as
different transactions -- do them at different times.
Thanks,
Shawn