You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ezequiel Calderara <ez...@gmail.com> on 2010/10/18 20:46:50 UTC
Commits on service after shutdown
Hi, i'm new in the mailing list.
I'm implementing Solr in my actual job, and i'm having some problems.
I was testing the consistency of the "commits". I found for example that if
we add X documents to the index (without commiting) and then we restart the
service, the documents are commited. They show up in the results. This is
interpreted to me like an error.
But when we add X documents to the index (without commiting) and then we
kill the process and we start it again, the documents doesn't appear. This
behaviour is the one i want.
Is there any param to avoid the auto-committing of documents after a
shutdown?
Is there any param to keep those un-commited documents "alive" after a kill?
Thanks!
--
______
Ezequiel.
Http://www.ironicnet.com <http://www.ironicnet.com/>
Re: Commits on service after shutdown
Posted by Jan Høydahl / Cominvent <ja...@cominvent.com>.
You never get full control of commits, as Solr will auto-commit anyway whenever the (configurable) input buffer is full. With the current architecture you cannot really trust adds or commits to 100% certainly be successful, because the server may have been restarted between an add and commit() without you noticing etc. So your feeder app should expect failures to happen, including added docs to be committed, and be prepared to re-sumbit any documents needed after a failure. This can be achieved by querying the index at regular intervals to see if you are in sync. Or you could help implement SOLR-1924 to get a reliable callback mechanism :)
--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com
On 18. okt. 2010, at 21.50, Ezequiel Calderara wrote:
> I understand, but i want to have control of what is commit or not.
> In our scenario, we want to add documents to the index, and maybe after an
> hour trigger the commit.
>
> If in the middle, we have a server shutdown or any process sending a
> Shutdown signal to the process. I don't want those documents being commited.
>
> Should i file a bug issue or an enhacement issue?.
>
> Thanks
>
>
> On Mon, Oct 18, 2010 at 3:54 PM, Israel Ekpo <is...@gmail.com> wrote:
>
>> The documents should be implicitly committed when the Lucene index is
>> closed.
>>
>> When you perform a graceful shutdown, the Lucene index gets closed and the
>> documents get committed implicitly.
>>
>> When the shutdown is abrupt as in a KILL -9, then this does not happen and
>> the updates are lost.
>>
>> You can use the auto commit parameter when sending your updates so that the
>> changes are saved right away, thought this could slow down the indexing
>> speed considerably but I do not believe there are parameters to keep those
>> un-commited documents "alive" after a kill.
>>
>>
>>
>> On Mon, Oct 18, 2010 at 2:46 PM, Ezequiel Calderara <ezechico@gmail.com
>>> wrote:
>>
>>> Hi, i'm new in the mailing list.
>>> I'm implementing Solr in my actual job, and i'm having some problems.
>>> I was testing the consistency of the "commits". I found for example that
>> if
>>> we add X documents to the index (without commiting) and then we restart
>> the
>>> service, the documents are commited. They show up in the results. This is
>>> interpreted to me like an error.
>>> But when we add X documents to the index (without commiting) and then we
>>> kill the process and we start it again, the documents doesn't appear.
>> This
>>> behaviour is the one i want.
>>>
>>> Is there any param to avoid the auto-committing of documents after a
>>> shutdown?
>>> Is there any param to keep those un-commited documents "alive" after a
>>> kill?
>>>
>>> Thanks!
>>>
>>> --
>>> ______
>>> Ezequiel.
>>>
>>> Http://www.ironicnet.com <http://www.ironicnet.com/> <
>> http://www.ironicnet.com/>
>>>
>>
>>
>>
>> --
>> °O°
>> "Good Enough" is not good enough.
>> To give anything less than your best is to sacrifice the gift.
>> Quality First. Measure Twice. Cut Once.
>> http://www.israelekpo.com/
>>
>
>
>
> --
> ______
> Ezequiel.
>
> Http://www.ironicnet.com
Re: Commits on service after shutdown
Posted by Ezequiel Calderara <ez...@gmail.com>.
I'll see if i can resolve this adding an extra core with the same schema for
holding this documents.
So, Core0 will act as a "Queue" and the Core1 will be the real index. And
the commit in the core0 will trigger an add to the core1 and its commit.
That way i can be sure of not losing data.
It surprises me that solr doesn't have this feature built-in. I still have
to verify the perfomance, but looks good to me.
Anyway, any help would be appreciated.
On Mon, Oct 18, 2010 at 5:05 PM, Ezequiel Calderara <ez...@gmail.com>wrote:
> But if something happens in between that hour, i will have lost or
> committed the documents to the index out of the schedule.
>
> How can i handle this scenario?
>
> I think that Solr (or Lucene) should make sure of the durability<http://en.wikipedia.org/wiki/Durability_(database_systems)>of the data even if its in an uncommited state.
> On Mon, Oct 18, 2010 at 4:53 PM, Matthew Hall <mhall@informatics.jax.org
> > wrote:
>
>> No.. you would just turn autocommit off, and have the thread that is
>> doing updates to your indexes commit every hour. I'd think that this would
>> take care of the scenario that you are describing.
>>
>> Matt
>>
>>
>> On 10/18/2010 3:50 PM, Ezequiel Calderara wrote:
>>
>>> I understand, but i want to have control of what is commit or not.
>>> In our scenario, we want to add documents to the index, and maybe after
>>> an
>>> hour trigger the commit.
>>>
>>> If in the middle, we have a server shutdown or any process sending a
>>> Shutdown signal to the process. I don't want those documents being
>>> commited.
>>>
>>> Should i file a bug issue or an enhacement issue?.
>>>
>>> Thanks
>>>
>>>
>>> On Mon, Oct 18, 2010 at 3:54 PM, Israel Ekpo<is...@gmail.com>
>>> wrote:
>>>
>>> The documents should be implicitly committed when the Lucene index is
>>>> closed.
>>>>
>>>> When you perform a graceful shutdown, the Lucene index gets closed and
>>>> the
>>>> documents get committed implicitly.
>>>>
>>>> When the shutdown is abrupt as in a KILL -9, then this does not happen
>>>> and
>>>> the updates are lost.
>>>>
>>>> You can use the auto commit parameter when sending your updates so that
>>>> the
>>>> changes are saved right away, thought this could slow down the indexing
>>>> speed considerably but I do not believe there are parameters to keep
>>>> those
>>>> un-commited documents "alive" after a kill.
>>>>
>>>>
>>>>
>>>> On Mon, Oct 18, 2010 at 2:46 PM, Ezequiel Calderara<ezechico@gmail.com
>>>>
>>>>> wrote:
>>>>> Hi, i'm new in the mailing list.
>>>>> I'm implementing Solr in my actual job, and i'm having some problems.
>>>>> I was testing the consistency of the "commits". I found for example
>>>>> that
>>>>>
>>>> if
>>>>
>>>>> we add X documents to the index (without commiting) and then we restart
>>>>>
>>>> the
>>>>
>>>>> service, the documents are commited. They show up in the results. This
>>>>> is
>>>>> interpreted to me like an error.
>>>>> But when we add X documents to the index (without commiting) and then
>>>>> we
>>>>> kill the process and we start it again, the documents doesn't appear.
>>>>>
>>>> This
>>>>
>>>>> behaviour is the one i want.
>>>>>
>>>>> Is there any param to avoid the auto-committing of documents after a
>>>>> shutdown?
>>>>> Is there any param to keep those un-commited documents "alive" after a
>>>>> kill?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> --
>>>>> ______
>>>>> Ezequiel.
>>>>>
>>>>> Http://www.ironicnet.com <http://www.ironicnet.com/><
>>>>> http://www.ironicnet.com/> <
>>>>>
>>>> http://www.ironicnet.com/>
>>>>
>>>>
>>>> --
>>>> °O°
>>>> "Good Enough" is not good enough.
>>>> To give anything less than your best is to sacrifice the gift.
>>>> Quality First. Measure Twice. Cut Once.
>>>> http://www.israelekpo.com/
>>>>
>>>>
>>>
>>>
>>
>
>
> --
> ______
> Ezequiel.
>
> Http://www.ironicnet.com <http://www.ironicnet.com/>
>
--
______
Ezequiel.
Http://www.ironicnet.com
Re: Commits on service after shutdown
Posted by Ezequiel Calderara <ez...@gmail.com>.
But if something happens in between that hour, i will have lost or committed
the documents to the index out of the schedule.
How can i handle this scenario?
I think that Solr (or Lucene) should make sure of the
durability<http://en.wikipedia.org/wiki/Durability_(database_systems)>of
the data even if its in an uncommited state.
On Mon, Oct 18, 2010 at 4:53 PM, Matthew Hall <mh...@informatics.jax.org>wrote:
> No.. you would just turn autocommit off, and have the thread that is doing
> updates to your indexes commit every hour. I'd think that this would take
> care of the scenario that you are describing.
>
> Matt
>
>
> On 10/18/2010 3:50 PM, Ezequiel Calderara wrote:
>
>> I understand, but i want to have control of what is commit or not.
>> In our scenario, we want to add documents to the index, and maybe after an
>> hour trigger the commit.
>>
>> If in the middle, we have a server shutdown or any process sending a
>> Shutdown signal to the process. I don't want those documents being
>> commited.
>>
>> Should i file a bug issue or an enhacement issue?.
>>
>> Thanks
>>
>>
>> On Mon, Oct 18, 2010 at 3:54 PM, Israel Ekpo<is...@gmail.com>
>> wrote:
>>
>> The documents should be implicitly committed when the Lucene index is
>>> closed.
>>>
>>> When you perform a graceful shutdown, the Lucene index gets closed and
>>> the
>>> documents get committed implicitly.
>>>
>>> When the shutdown is abrupt as in a KILL -9, then this does not happen
>>> and
>>> the updates are lost.
>>>
>>> You can use the auto commit parameter when sending your updates so that
>>> the
>>> changes are saved right away, thought this could slow down the indexing
>>> speed considerably but I do not believe there are parameters to keep
>>> those
>>> un-commited documents "alive" after a kill.
>>>
>>>
>>>
>>> On Mon, Oct 18, 2010 at 2:46 PM, Ezequiel Calderara<ezechico@gmail.com
>>>
>>>> wrote:
>>>> Hi, i'm new in the mailing list.
>>>> I'm implementing Solr in my actual job, and i'm having some problems.
>>>> I was testing the consistency of the "commits". I found for example that
>>>>
>>> if
>>>
>>>> we add X documents to the index (without commiting) and then we restart
>>>>
>>> the
>>>
>>>> service, the documents are commited. They show up in the results. This
>>>> is
>>>> interpreted to me like an error.
>>>> But when we add X documents to the index (without commiting) and then we
>>>> kill the process and we start it again, the documents doesn't appear.
>>>>
>>> This
>>>
>>>> behaviour is the one i want.
>>>>
>>>> Is there any param to avoid the auto-committing of documents after a
>>>> shutdown?
>>>> Is there any param to keep those un-commited documents "alive" after a
>>>> kill?
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>> ______
>>>> Ezequiel.
>>>>
>>>> Http://www.ironicnet.com <http://www.ironicnet.com/><
>>>> http://www.ironicnet.com/> <
>>>>
>>> http://www.ironicnet.com/>
>>>
>>>
>>> --
>>> °O°
>>> "Good Enough" is not good enough.
>>> To give anything less than your best is to sacrifice the gift.
>>> Quality First. Measure Twice. Cut Once.
>>> http://www.israelekpo.com/
>>>
>>>
>>
>>
>
--
______
Ezequiel.
Http://www.ironicnet.com
Re: Commits on service after shutdown
Posted by Matthew Hall <mh...@informatics.jax.org>.
No.. you would just turn autocommit off, and have the thread that is
doing updates to your indexes commit every hour. I'd think that this
would take care of the scenario that you are describing.
Matt
On 10/18/2010 3:50 PM, Ezequiel Calderara wrote:
> I understand, but i want to have control of what is commit or not.
> In our scenario, we want to add documents to the index, and maybe after an
> hour trigger the commit.
>
> If in the middle, we have a server shutdown or any process sending a
> Shutdown signal to the process. I don't want those documents being commited.
>
> Should i file a bug issue or an enhacement issue?.
>
> Thanks
>
>
> On Mon, Oct 18, 2010 at 3:54 PM, Israel Ekpo<is...@gmail.com> wrote:
>
>> The documents should be implicitly committed when the Lucene index is
>> closed.
>>
>> When you perform a graceful shutdown, the Lucene index gets closed and the
>> documents get committed implicitly.
>>
>> When the shutdown is abrupt as in a KILL -9, then this does not happen and
>> the updates are lost.
>>
>> You can use the auto commit parameter when sending your updates so that the
>> changes are saved right away, thought this could slow down the indexing
>> speed considerably but I do not believe there are parameters to keep those
>> un-commited documents "alive" after a kill.
>>
>>
>>
>> On Mon, Oct 18, 2010 at 2:46 PM, Ezequiel Calderara<ezechico@gmail.com
>>> wrote:
>>> Hi, i'm new in the mailing list.
>>> I'm implementing Solr in my actual job, and i'm having some problems.
>>> I was testing the consistency of the "commits". I found for example that
>> if
>>> we add X documents to the index (without commiting) and then we restart
>> the
>>> service, the documents are commited. They show up in the results. This is
>>> interpreted to me like an error.
>>> But when we add X documents to the index (without commiting) and then we
>>> kill the process and we start it again, the documents doesn't appear.
>> This
>>> behaviour is the one i want.
>>>
>>> Is there any param to avoid the auto-committing of documents after a
>>> shutdown?
>>> Is there any param to keep those un-commited documents "alive" after a
>>> kill?
>>>
>>> Thanks!
>>>
>>> --
>>> ______
>>> Ezequiel.
>>>
>>> Http://www.ironicnet.com<http://www.ironicnet.com/> <
>> http://www.ironicnet.com/>
>>
>>
>> --
>> °O°
>> "Good Enough" is not good enough.
>> To give anything less than your best is to sacrifice the gift.
>> Quality First. Measure Twice. Cut Once.
>> http://www.israelekpo.com/
>>
>
>
Re: Commits on service after shutdown
Posted by Ezequiel Calderara <ez...@gmail.com>.
I understand, but i want to have control of what is commit or not.
In our scenario, we want to add documents to the index, and maybe after an
hour trigger the commit.
If in the middle, we have a server shutdown or any process sending a
Shutdown signal to the process. I don't want those documents being commited.
Should i file a bug issue or an enhacement issue?.
Thanks
On Mon, Oct 18, 2010 at 3:54 PM, Israel Ekpo <is...@gmail.com> wrote:
> The documents should be implicitly committed when the Lucene index is
> closed.
>
> When you perform a graceful shutdown, the Lucene index gets closed and the
> documents get committed implicitly.
>
> When the shutdown is abrupt as in a KILL -9, then this does not happen and
> the updates are lost.
>
> You can use the auto commit parameter when sending your updates so that the
> changes are saved right away, thought this could slow down the indexing
> speed considerably but I do not believe there are parameters to keep those
> un-commited documents "alive" after a kill.
>
>
>
> On Mon, Oct 18, 2010 at 2:46 PM, Ezequiel Calderara <ezechico@gmail.com
> >wrote:
>
> > Hi, i'm new in the mailing list.
> > I'm implementing Solr in my actual job, and i'm having some problems.
> > I was testing the consistency of the "commits". I found for example that
> if
> > we add X documents to the index (without commiting) and then we restart
> the
> > service, the documents are commited. They show up in the results. This is
> > interpreted to me like an error.
> > But when we add X documents to the index (without commiting) and then we
> > kill the process and we start it again, the documents doesn't appear.
> This
> > behaviour is the one i want.
> >
> > Is there any param to avoid the auto-committing of documents after a
> > shutdown?
> > Is there any param to keep those un-commited documents "alive" after a
> > kill?
> >
> > Thanks!
> >
> > --
> > ______
> > Ezequiel.
> >
> > Http://www.ironicnet.com <http://www.ironicnet.com/> <
> http://www.ironicnet.com/>
> >
>
>
>
> --
> °O°
> "Good Enough" is not good enough.
> To give anything less than your best is to sacrifice the gift.
> Quality First. Measure Twice. Cut Once.
> http://www.israelekpo.com/
>
--
______
Ezequiel.
Http://www.ironicnet.com
Re: Commits on service after shutdown
Posted by Israel Ekpo <is...@gmail.com>.
The documents should be implicitly committed when the Lucene index is
closed.
When you perform a graceful shutdown, the Lucene index gets closed and the
documents get committed implicitly.
When the shutdown is abrupt as in a KILL -9, then this does not happen and
the updates are lost.
You can use the auto commit parameter when sending your updates so that the
changes are saved right away, thought this could slow down the indexing
speed considerably but I do not believe there are parameters to keep those
un-commited documents "alive" after a kill.
On Mon, Oct 18, 2010 at 2:46 PM, Ezequiel Calderara <ez...@gmail.com>wrote:
> Hi, i'm new in the mailing list.
> I'm implementing Solr in my actual job, and i'm having some problems.
> I was testing the consistency of the "commits". I found for example that if
> we add X documents to the index (without commiting) and then we restart the
> service, the documents are commited. They show up in the results. This is
> interpreted to me like an error.
> But when we add X documents to the index (without commiting) and then we
> kill the process and we start it again, the documents doesn't appear. This
> behaviour is the one i want.
>
> Is there any param to avoid the auto-committing of documents after a
> shutdown?
> Is there any param to keep those un-commited documents "alive" after a
> kill?
>
> Thanks!
>
> --
> ______
> Ezequiel.
>
> Http://www.ironicnet.com <http://www.ironicnet.com/>
>
--
°O°
"Good Enough" is not good enough.
To give anything less than your best is to sacrifice the gift.
Quality First. Measure Twice. Cut Once.
http://www.israelekpo.com/