You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Josh Turmel <jt...@gmail.com> on 2012/11/06 22:01:19 UTC
Deleting an individual document while delta index is running
Running Solr 3.3
We're running into issues where deleting individual documents (by ID) will
timeout but it only seems to happen when our hourly delta index is being
ran to pull in new documents, is there a way to work around this?
Thank you,
Josh
Re: Deleting an individual document while delta index is
running
Posted by Josh Turmel <jt...@gmail.com>.
Okay, thanks for the help guys... I *think* that this can be resolved by kicking off the delta and passing optimize=false since the default was true in 3.3.
I'll post back if I see the issue pop back up.
JT
On Wednesday, November 7, 2012 at 1:34 PM, Josh Turmel wrote:
> Here's what we have set in our data-config.xml
>
> <dataSource name="jdbc" driver="org.postgresql.Driver" url="jdbc:postgresql://localhost:5432/reader" user="data"
> batchSize="1000" readOnly="true" autoCommit="false"
> transactionIsolation="TRANSACTION_READ_COMMITTED" holdability="CLOSE_CURSORS_AT_COMMIT"
> />
>
>
> Thanks,
> Josh Turmel
>
>
> On Wednesday, November 7, 2012 at 1:00 PM, Shawn Heisey wrote:
>
> > On 11/7/2012 10:55 AM, Otis Gospodnetic wrote:
> > > Hi Shawn,
> > >
> > > It the last part really correct? Optimization should be doable while
> > > updates are going on... or am I missing something?
> > >
> >
> >
> > From what I recall when I was first putting my build system together,
> > which I will admit was on Solr 1.4.0, I couldn't do updates/deletes
> > while optimizing was underway. I don't think 3.x was a whole lot
> > different in this respect. From the little I understand about the
> > significant changes in 4.0, it is probably now possible to do everything
> > at the same time with no worry.
> >
> > Because they are using 3.3, I don't think they have access to this
> > ability. Given the limited amount of information available, it seemed
> > the most likely explanation. I could be wrong, and if I am, they will
> > have to keep looking for an explanation.
> >
> > I'm definitely no expert, and I have not tried optimizing and updating
> > at the same time since upgrading to 3.x. My indexing system
> > deliberately avoids doing the two at the same time because it caused
> > problems on 1.4.x.
> >
> > I would certainly love to know for sure whether it's possible on 4.0,
> > because I am in the process of updating my entire test environment in
> > preparation for a production rollout. If I can do updates/commits at
> > the same time as optimizing, my code will get smaller and a lot simpler.
> >
> > Thanks,
> > Shawn
> >
> >
> >
>
>
Re: Deleting an individual document while delta index is
running
Posted by Josh Turmel <jt...@gmail.com>.
Here's what we have set in our data-config.xml
<dataSource name="jdbc" driver="org.postgresql.Driver" url="jdbc:postgresql://localhost:5432/reader" user="data"
batchSize="1000" readOnly="true" autoCommit="false"
transactionIsolation="TRANSACTION_READ_COMMITTED" holdability="CLOSE_CURSORS_AT_COMMIT"
/>
Thanks,
Josh Turmel
On Wednesday, November 7, 2012 at 1:00 PM, Shawn Heisey wrote:
> On 11/7/2012 10:55 AM, Otis Gospodnetic wrote:
> > Hi Shawn,
> >
> > It the last part really correct? Optimization should be doable while
> > updates are going on... or am I missing something?
> >
>
>
> From what I recall when I was first putting my build system together,
> which I will admit was on Solr 1.4.0, I couldn't do updates/deletes
> while optimizing was underway. I don't think 3.x was a whole lot
> different in this respect. From the little I understand about the
> significant changes in 4.0, it is probably now possible to do everything
> at the same time with no worry.
>
> Because they are using 3.3, I don't think they have access to this
> ability. Given the limited amount of information available, it seemed
> the most likely explanation. I could be wrong, and if I am, they will
> have to keep looking for an explanation.
>
> I'm definitely no expert, and I have not tried optimizing and updating
> at the same time since upgrading to 3.x. My indexing system
> deliberately avoids doing the two at the same time because it caused
> problems on 1.4.x.
>
> I would certainly love to know for sure whether it's possible on 4.0,
> because I am in the process of updating my entire test environment in
> preparation for a production rollout. If I can do updates/commits at
> the same time as optimizing, my code will get smaller and a lot simpler.
>
> Thanks,
> Shawn
>
>
Re: Deleting an individual document while delta index is running
Posted by Shawn Heisey <so...@elyograg.org>.
On 11/7/2012 10:55 AM, Otis Gospodnetic wrote:
> Hi Shawn,
>
> It the last part really correct? Optimization should be doable while
> updates are going on... or am I missing something?
From what I recall when I was first putting my build system together,
which I will admit was on Solr 1.4.0, I couldn't do updates/deletes
while optimizing was underway. I don't think 3.x was a whole lot
different in this respect. From the little I understand about the
significant changes in 4.0, it is probably now possible to do everything
at the same time with no worry.
Because they are using 3.3, I don't think they have access to this
ability. Given the limited amount of information available, it seemed
the most likely explanation. I could be wrong, and if I am, they will
have to keep looking for an explanation.
I'm definitely no expert, and I have not tried optimizing and updating
at the same time since upgrading to 3.x. My indexing system
deliberately avoids doing the two at the same time because it caused
problems on 1.4.x.
I would certainly love to know for sure whether it's possible on 4.0,
because I am in the process of updating my entire test environment in
preparation for a production rollout. If I can do updates/commits at
the same time as optimizing, my code will get smaller and a lot simpler.
Thanks,
Shawn
Re: Deleting an individual document while delta index is running
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi Shawn,
It the last part really correct? Optimization should be doable while
updates are going on... or am I missing something?
Otis
--
Search Analytics - http://sematext.com/search-analytics/index.html
Performance Monitoring - http://sematext.com/spm/index.html
On Wed, Nov 7, 2012 at 12:17 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 11/6/2012 2:01 PM, Josh Turmel wrote:
>
>> Running Solr 3.3
>>
>> We're running into issues where deleting individual documents (by ID) will
>> timeout but it only seems to happen when our hourly delta index is being
>> ran to pull in new documents, is there a way to work around this?
>>
>
> Are you experiencing really long commit times with your delta indexing, or
> are you running an optimize after the delta index? That's the only thing I
> can think of that might cause problems like that.
>
> General recommendations: 1) Reduce commit times. Large autowarm counts on
> the Solr caches will make commits very slow. Also, make sure you've got
> enough RAM so your OS disk cache runs efficiently. 2) Don't optimize
> often, unless the index is really really small and it can complete quickly.
> On an index that never sees document deletes or re-indexes, optimizing is
> completely unnecessary. Keep in mind that re-indexing an existing document
> does an implicit delete first. Since you are deleting documents, you'll
> want to do an optimize occasionally to keep your index clean, but you
> should do it infrequently - once a day or similar.
>
> Another critical note about optimizing: No matter how often you do it,
> you'll want to make sure that no other updates are happening on that index
> while it is optimizing. Since I know this may be the obvious next
> question: I am not aware of any way to query Solr and ask it if an optimize
> is underway. You'll need to coordinate it yourself.
>
> Thanks,
> Shawn
>
>
Re: Deleting an individual document while delta index is running
Posted by Shawn Heisey <so...@elyograg.org>.
On 11/6/2012 2:01 PM, Josh Turmel wrote:
> Running Solr 3.3
>
> We're running into issues where deleting individual documents (by ID) will
> timeout but it only seems to happen when our hourly delta index is being
> ran to pull in new documents, is there a way to work around this?
Are you experiencing really long commit times with your delta indexing,
or are you running an optimize after the delta index? That's the only
thing I can think of that might cause problems like that.
General recommendations: 1) Reduce commit times. Large autowarm counts
on the Solr caches will make commits very slow. Also, make sure you've
got enough RAM so your OS disk cache runs efficiently. 2) Don't
optimize often, unless the index is really really small and it can
complete quickly. On an index that never sees document deletes or
re-indexes, optimizing is completely unnecessary. Keep in mind that
re-indexing an existing document does an implicit delete first. Since
you are deleting documents, you'll want to do an optimize occasionally
to keep your index clean, but you should do it infrequently - once a day
or similar.
Another critical note about optimizing: No matter how often you do it,
you'll want to make sure that no other updates are happening on that
index while it is optimizing. Since I know this may be the obvious next
question: I am not aware of any way to query Solr and ask it if an
optimize is underway. You'll need to coordinate it yourself.
Thanks,
Shawn