You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Mike Leddy <mi...@loop.com.br> on 2011/09/13 23:39:04 UTC
View group compactions shutting down in trunk
Hello,
I've been experimenting with trunk and the new compaction daemon,
and discovered that some of my larger views were never being fully
compacted.
Basically I am encountering a large number of these situations:
[Tue, 13 Sep 2011 13:24:46 GMT] [info] [<0.173.0>] Shutting down view group server, monitored db is closing.
[Tue, 13 Sep 2011 13:24:46 GMT] [error] [<0.155.0>] Compaction daemon - an error ocurred while compacting the view group `base` from database `XXXX-1301529600-1303948799`: shutdown
I suspect that since I have a large number of other smaller databases
(approx 2500) that are constantly being updated there is a constant
stream of databases being closed as a result of least recently used
recovery - I currently have max_open_dbs at 500.
This does not affect database compaction as a database is never
considered idle when compacting.
Consequently the larger views are shutdown before completion. Also,
when a new attempt to re-compact is processed the partially complete
*.compact.view file is truncated. As a result these large views
never get fully compacted.
If the view group compaction continued where it left off I guess
the impact would be minimal.
Has anyone else seen this ?
Regards,
Mike
Re: View group compactions shutting down in trunk
Posted by Mike Leddy <mi...@loop.com.br>.
Thanks, Filipe. I applied the patch and it has resolved the problem for
me - I have been running this version since yesterday.
Here the view compaction took approx 20 minutes and completed
successfully even though database resources were being constantly
reclaimed by the lru mechanism.
[Thu, 15 Sep 2011 15:11:50 GMT] [info] [<0.30538.35>] View index compaction starting for XXXX-1306368000-1308787199 _design/admin
[Thu, 15 Sep 2011 15:30:32 GMT] [info] [<0.30538.35>] View index compaction complete for XXXX-1306368000-1308787199 _design/admin
[Thu, 15 Sep 2011 15:31:17 GMT] [info] [<0.30538.35>] Shutting down view group server, monitored db is closing.
regards,
Mike
On Wed, 2011-09-14 at 23:19 -0700, Filipe David Manana wrote:
> Mike, I was able to create a test case to reproduce your issue and
> validate the last patch. I created an issue in Jira for it:
>
> https://issues.apache.org/jira/browse/COUCHDB-1283
>
> regards,
>
> On Wed, Sep 14, 2011 at 8:04 AM, Filipe David Manana
> <fd...@apache.org> wrote:
> > Mike, this is the simplest approach I was talking about earlier, in
> > case you want to try it:
> >
> > http://friendpaste.com/1s0In2cLBhWcdsAWEqAw1E
> >
> > regards,
> >
> > On Wed, Sep 14, 2011 at 6:54 AM, Filipe David Manana
> > <fd...@apache.org> wrote:
> >> Makes sense Mike. I think the simplest is to make the view compactor keep
> >> the database open.
> >> I've been thinking in the meanwhile for other approaches however.
> >> I'll get back to this soon.
> >>
> >> Thanks for reporting this and testing
> >>
> >> On Wednesday, September 14, 2011, Mike Leddy <mi...@loop.com.br> wrote:
> >>>
> >>> Thanks for the patch Filipe, but I am pretty sure that the compaction
> >>> daemon is not involved in this.
> >>>
> >>> I removed all item in the 'compactions' config and restarted couchdb.
> >>> As ?CONFIG_ETS is empty that compact_loop does nothing and
> >>> maybe_compact_db is never called.
> >>>
> >>> Next I started a view compaction in Futon whilst doing the inserts
> >>> that cause the constant lru recovery. As expected the result was the
> >>> the same:
> >>>
> >>> "Shutting down view group server, monitored db is closing" in the
> >>> log at the same time the view group compaction was shut down.
> >>>
> >>> I had never seen this before as I had no reason to compact these
> >>> large views whilst having so many concurrent accesses on the
> >>> smaller databases.
> >>>
> >>> Regards,
> >>>
> >>> Mike
> >>>
> >>> On Tue, 2011-09-13 at 20:53 -0700, Filipe David Manana wrote:
> >>>> Mike,
> >>>>
> >>>> Can you try the following patch?
> >>>>
> >>>> http://friendpaste.com/2fT5Ne4NAQYoS9CWBsOKUW
> >>>>
> >>>> Basically it keeps the respective database open until view compaction
> >>>> finishes.
> >>>>
> >>>> Thanks for sharing this
> >>>>
> >>>> On Tue, Sep 13, 2011 at 2:39 PM, Mike Leddy <mi...@loop.com.br> wrote:
> >>>> > Hello,
> >>>> >
> >>>> > I've been experimenting with trunk and the new compaction daemon,
> >>>> > and discovered that some of my larger views were never being fully
> >>>> > compacted.
> >>>> >
> >>>> > Basically I am encountering a large number of these situations:
> >>>> >
> >>>> > [Tue, 13 Sep 2011 13:24:46 GMT] [info] [<0.173.0>] Shutting down view
> >>>> > group server, monitored db is closing.
> >>>> > [Tue, 13 Sep 2011 13:24:46 GMT] [error] [<0.155.0>] Compaction daemon -
> >>>> > an error ocurred while compacting the view group `base` from database
> >>>> > `XXXX-1301529600-1303948799`: shutdown
> >>>> >
> >>>> > I suspect that since I have a large number of other smaller databases
> >>>> > (approx 2500) that are constantly being updated there is a constant
> >>>> > stream of databases being closed as a result of least recently used
> >>>> > recovery - I currently have max_open_dbs at 500.
> >>>> >
> >>>> > This does not affect database compaction as a database is never
> >>>> > considered idle when compacting.
> >>>> >
> >>>> > Consequently the larger views are shutdown before completion. Also,
> >>>> > when a new attempt to re-compact is processed the partially complete
> >>>> > *.compact.view file is truncated. As a result these large views
> >>>> > never get fully compacted.
> >>>> >
> >>>> > If the view group compaction continued where it left off I guess
> >>>> > the impact would be minimal.
> >>>> >
> >>>> > Has anyone else seen this ?
> >>>> >
> >>>> > Regards,
> >>>> >
> >>>> > Mike
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >> --
> >> Filipe David Manana,
> >>
> >> "Reasonable men adapt themselves to the world.
> >> Unreasonable men adapt the world to themselves.
> >> That's why all progress depends on unreasonable men."
> >>
> >>
> >
> >
> >
> > --
> > Filipe David Manana,
> >
> > "Reasonable men adapt themselves to the world.
> > Unreasonable men adapt the world to themselves.
> > That's why all progress depends on unreasonable men."
> >
>
>
>
Re: View group compactions shutting down in trunk
Posted by Filipe David Manana <fd...@apache.org>.
Mike, I was able to create a test case to reproduce your issue and
validate the last patch. I created an issue in Jira for it:
https://issues.apache.org/jira/browse/COUCHDB-1283
regards,
On Wed, Sep 14, 2011 at 8:04 AM, Filipe David Manana
<fd...@apache.org> wrote:
> Mike, this is the simplest approach I was talking about earlier, in
> case you want to try it:
>
> http://friendpaste.com/1s0In2cLBhWcdsAWEqAw1E
>
> regards,
>
> On Wed, Sep 14, 2011 at 6:54 AM, Filipe David Manana
> <fd...@apache.org> wrote:
>> Makes sense Mike. I think the simplest is to make the view compactor keep
>> the database open.
>> I've been thinking in the meanwhile for other approaches however.
>> I'll get back to this soon.
>>
>> Thanks for reporting this and testing
>>
>> On Wednesday, September 14, 2011, Mike Leddy <mi...@loop.com.br> wrote:
>>>
>>> Thanks for the patch Filipe, but I am pretty sure that the compaction
>>> daemon is not involved in this.
>>>
>>> I removed all item in the 'compactions' config and restarted couchdb.
>>> As ?CONFIG_ETS is empty that compact_loop does nothing and
>>> maybe_compact_db is never called.
>>>
>>> Next I started a view compaction in Futon whilst doing the inserts
>>> that cause the constant lru recovery. As expected the result was the
>>> the same:
>>>
>>> "Shutting down view group server, monitored db is closing" in the
>>> log at the same time the view group compaction was shut down.
>>>
>>> I had never seen this before as I had no reason to compact these
>>> large views whilst having so many concurrent accesses on the
>>> smaller databases.
>>>
>>> Regards,
>>>
>>> Mike
>>>
>>> On Tue, 2011-09-13 at 20:53 -0700, Filipe David Manana wrote:
>>>> Mike,
>>>>
>>>> Can you try the following patch?
>>>>
>>>> http://friendpaste.com/2fT5Ne4NAQYoS9CWBsOKUW
>>>>
>>>> Basically it keeps the respective database open until view compaction
>>>> finishes.
>>>>
>>>> Thanks for sharing this
>>>>
>>>> On Tue, Sep 13, 2011 at 2:39 PM, Mike Leddy <mi...@loop.com.br> wrote:
>>>> > Hello,
>>>> >
>>>> > I've been experimenting with trunk and the new compaction daemon,
>>>> > and discovered that some of my larger views were never being fully
>>>> > compacted.
>>>> >
>>>> > Basically I am encountering a large number of these situations:
>>>> >
>>>> > [Tue, 13 Sep 2011 13:24:46 GMT] [info] [<0.173.0>] Shutting down view
>>>> > group server, monitored db is closing.
>>>> > [Tue, 13 Sep 2011 13:24:46 GMT] [error] [<0.155.0>] Compaction daemon -
>>>> > an error ocurred while compacting the view group `base` from database
>>>> > `XXXX-1301529600-1303948799`: shutdown
>>>> >
>>>> > I suspect that since I have a large number of other smaller databases
>>>> > (approx 2500) that are constantly being updated there is a constant
>>>> > stream of databases being closed as a result of least recently used
>>>> > recovery - I currently have max_open_dbs at 500.
>>>> >
>>>> > This does not affect database compaction as a database is never
>>>> > considered idle when compacting.
>>>> >
>>>> > Consequently the larger views are shutdown before completion. Also,
>>>> > when a new attempt to re-compact is processed the partially complete
>>>> > *.compact.view file is truncated. As a result these large views
>>>> > never get fully compacted.
>>>> >
>>>> > If the view group compaction continued where it left off I guess
>>>> > the impact would be minimal.
>>>> >
>>>> > Has anyone else seen this ?
>>>> >
>>>> > Regards,
>>>> >
>>>> > Mike
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>> --
>> Filipe David Manana,
>>
>> "Reasonable men adapt themselves to the world.
>> Unreasonable men adapt the world to themselves.
>> That's why all progress depends on unreasonable men."
>>
>>
>
>
>
> --
> Filipe David Manana,
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>
--
Filipe David Manana,
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: View group compactions shutting down in trunk
Posted by Filipe David Manana <fd...@apache.org>.
Mike, this is the simplest approach I was talking about earlier, in
case you want to try it:
http://friendpaste.com/1s0In2cLBhWcdsAWEqAw1E
regards,
On Wed, Sep 14, 2011 at 6:54 AM, Filipe David Manana
<fd...@apache.org> wrote:
> Makes sense Mike. I think the simplest is to make the view compactor keep
> the database open.
> I've been thinking in the meanwhile for other approaches however.
> I'll get back to this soon.
>
> Thanks for reporting this and testing
>
> On Wednesday, September 14, 2011, Mike Leddy <mi...@loop.com.br> wrote:
>>
>> Thanks for the patch Filipe, but I am pretty sure that the compaction
>> daemon is not involved in this.
>>
>> I removed all item in the 'compactions' config and restarted couchdb.
>> As ?CONFIG_ETS is empty that compact_loop does nothing and
>> maybe_compact_db is never called.
>>
>> Next I started a view compaction in Futon whilst doing the inserts
>> that cause the constant lru recovery. As expected the result was the
>> the same:
>>
>> "Shutting down view group server, monitored db is closing" in the
>> log at the same time the view group compaction was shut down.
>>
>> I had never seen this before as I had no reason to compact these
>> large views whilst having so many concurrent accesses on the
>> smaller databases.
>>
>> Regards,
>>
>> Mike
>>
>> On Tue, 2011-09-13 at 20:53 -0700, Filipe David Manana wrote:
>>> Mike,
>>>
>>> Can you try the following patch?
>>>
>>> http://friendpaste.com/2fT5Ne4NAQYoS9CWBsOKUW
>>>
>>> Basically it keeps the respective database open until view compaction
>>> finishes.
>>>
>>> Thanks for sharing this
>>>
>>> On Tue, Sep 13, 2011 at 2:39 PM, Mike Leddy <mi...@loop.com.br> wrote:
>>> > Hello,
>>> >
>>> > I've been experimenting with trunk and the new compaction daemon,
>>> > and discovered that some of my larger views were never being fully
>>> > compacted.
>>> >
>>> > Basically I am encountering a large number of these situations:
>>> >
>>> > [Tue, 13 Sep 2011 13:24:46 GMT] [info] [<0.173.0>] Shutting down view
>>> > group server, monitored db is closing.
>>> > [Tue, 13 Sep 2011 13:24:46 GMT] [error] [<0.155.0>] Compaction daemon -
>>> > an error ocurred while compacting the view group `base` from database
>>> > `XXXX-1301529600-1303948799`: shutdown
>>> >
>>> > I suspect that since I have a large number of other smaller databases
>>> > (approx 2500) that are constantly being updated there is a constant
>>> > stream of databases being closed as a result of least recently used
>>> > recovery - I currently have max_open_dbs at 500.
>>> >
>>> > This does not affect database compaction as a database is never
>>> > considered idle when compacting.
>>> >
>>> > Consequently the larger views are shutdown before completion. Also,
>>> > when a new attempt to re-compact is processed the partially complete
>>> > *.compact.view file is truncated. As a result these large views
>>> > never get fully compacted.
>>> >
>>> > If the view group compaction continued where it left off I guess
>>> > the impact would be minimal.
>>> >
>>> > Has anyone else seen this ?
>>> >
>>> > Regards,
>>> >
>>> > Mike
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>>
>>
>>
>>
>
> --
> Filipe David Manana,
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>
>
--
Filipe David Manana,
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: View group compactions shutting down in trunk
Posted by Filipe David Manana <fd...@apache.org>.
Makes sense Mike. I think the simplest is to make the view compactor keep
the database open.
I've been thinking in the meanwhile for other approaches however.
I'll get back to this soon.
Thanks for reporting this and testing
On Wednesday, September 14, 2011, Mike Leddy <mi...@loop.com.br> wrote:
>
> Thanks for the patch Filipe, but I am pretty sure that the compaction
> daemon is not involved in this.
>
> I removed all item in the 'compactions' config and restarted couchdb.
> As ?CONFIG_ETS is empty that compact_loop does nothing and
> maybe_compact_db is never called.
>
> Next I started a view compaction in Futon whilst doing the inserts
> that cause the constant lru recovery. As expected the result was the
> the same:
>
> "Shutting down view group server, monitored db is closing" in the
> log at the same time the view group compaction was shut down.
>
> I had never seen this before as I had no reason to compact these
> large views whilst having so many concurrent accesses on the
> smaller databases.
>
> Regards,
>
> Mike
>
> On Tue, 2011-09-13 at 20:53 -0700, Filipe David Manana wrote:
>> Mike,
>>
>> Can you try the following patch?
>>
>> http://friendpaste.com/2fT5Ne4NAQYoS9CWBsOKUW
>>
>> Basically it keeps the respective database open until view compaction
finishes.
>>
>> Thanks for sharing this
>>
>> On Tue, Sep 13, 2011 at 2:39 PM, Mike Leddy <mi...@loop.com.br> wrote:
>> > Hello,
>> >
>> > I've been experimenting with trunk and the new compaction daemon,
>> > and discovered that some of my larger views were never being fully
>> > compacted.
>> >
>> > Basically I am encountering a large number of these situations:
>> >
>> > [Tue, 13 Sep 2011 13:24:46 GMT] [info] [<0.173.0>] Shutting down view
group server, monitored db is closing.
>> > [Tue, 13 Sep 2011 13:24:46 GMT] [error] [<0.155.0>] Compaction daemon -
an error ocurred while compacting the view group `base` from database
`XXXX-1301529600-1303948799`: shutdown
>> >
>> > I suspect that since I have a large number of other smaller databases
>> > (approx 2500) that are constantly being updated there is a constant
>> > stream of databases being closed as a result of least recently used
>> > recovery - I currently have max_open_dbs at 500.
>> >
>> > This does not affect database compaction as a database is never
>> > considered idle when compacting.
>> >
>> > Consequently the larger views are shutdown before completion. Also,
>> > when a new attempt to re-compact is processed the partially complete
>> > *.compact.view file is truncated. As a result these large views
>> > never get fully compacted.
>> >
>> > If the view group compaction continued where it left off I guess
>> > the impact would be minimal.
>> >
>> > Has anyone else seen this ?
>> >
>> > Regards,
>> >
>> > Mike
>> >
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>
>
>
--
Filipe David Manana,
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: View group compactions shutting down in trunk
Posted by Mike Leddy <mi...@loop.com.br>.
Thanks for the patch Filipe, but I am pretty sure that the compaction
daemon is not involved in this.
I removed all item in the 'compactions' config and restarted couchdb.
As ?CONFIG_ETS is empty that compact_loop does nothing and
maybe_compact_db is never called.
Next I started a view compaction in Futon whilst doing the inserts
that cause the constant lru recovery. As expected the result was the
the same:
"Shutting down view group server, monitored db is closing" in the
log at the same time the view group compaction was shut down.
I had never seen this before as I had no reason to compact these
large views whilst having so many concurrent accesses on the
smaller databases.
Regards,
Mike
On Tue, 2011-09-13 at 20:53 -0700, Filipe David Manana wrote:
> Mike,
>
> Can you try the following patch?
>
> http://friendpaste.com/2fT5Ne4NAQYoS9CWBsOKUW
>
> Basically it keeps the respective database open until view compaction finishes.
>
> Thanks for sharing this
>
> On Tue, Sep 13, 2011 at 2:39 PM, Mike Leddy <mi...@loop.com.br> wrote:
> > Hello,
> >
> > I've been experimenting with trunk and the new compaction daemon,
> > and discovered that some of my larger views were never being fully
> > compacted.
> >
> > Basically I am encountering a large number of these situations:
> >
> > [Tue, 13 Sep 2011 13:24:46 GMT] [info] [<0.173.0>] Shutting down view group server, monitored db is closing.
> > [Tue, 13 Sep 2011 13:24:46 GMT] [error] [<0.155.0>] Compaction daemon - an error ocurred while compacting the view group `base` from database `XXXX-1301529600-1303948799`: shutdown
> >
> > I suspect that since I have a large number of other smaller databases
> > (approx 2500) that are constantly being updated there is a constant
> > stream of databases being closed as a result of least recently used
> > recovery - I currently have max_open_dbs at 500.
> >
> > This does not affect database compaction as a database is never
> > considered idle when compacting.
> >
> > Consequently the larger views are shutdown before completion. Also,
> > when a new attempt to re-compact is processed the partially complete
> > *.compact.view file is truncated. As a result these large views
> > never get fully compacted.
> >
> > If the view group compaction continued where it left off I guess
> > the impact would be minimal.
> >
> > Has anyone else seen this ?
> >
> > Regards,
> >
> > Mike
> >
> >
> >
> >
> >
> >
>
>
>
Re: View group compactions shutting down in trunk
Posted by Paul Davis <pa...@gmail.com>.
That seems like a reasonable patch but I'm confused why we never had
reports of this before. Is it just that we didn't have people cycling
through db's fast enough that noticed view compaction dying or has
something else changed? I'd consider it unlikely that no one would've
noticed by now, but not unsurprising.
On Wed, Sep 14, 2011 at 3:53 AM, Filipe David Manana
<fd...@apache.org> wrote:
> Mike,
>
> Can you try the following patch?
>
> http://friendpaste.com/2fT5Ne4NAQYoS9CWBsOKUW
>
> Basically it keeps the respective database open until view compaction finishes.
>
> Thanks for sharing this
>
> On Tue, Sep 13, 2011 at 2:39 PM, Mike Leddy <mi...@loop.com.br> wrote:
>> Hello,
>>
>> I've been experimenting with trunk and the new compaction daemon,
>> and discovered that some of my larger views were never being fully
>> compacted.
>>
>> Basically I am encountering a large number of these situations:
>>
>> [Tue, 13 Sep 2011 13:24:46 GMT] [info] [<0.173.0>] Shutting down view group server, monitored db is closing.
>> [Tue, 13 Sep 2011 13:24:46 GMT] [error] [<0.155.0>] Compaction daemon - an error ocurred while compacting the view group `base` from database `XXXX-1301529600-1303948799`: shutdown
>>
>> I suspect that since I have a large number of other smaller databases
>> (approx 2500) that are constantly being updated there is a constant
>> stream of databases being closed as a result of least recently used
>> recovery - I currently have max_open_dbs at 500.
>>
>> This does not affect database compaction as a database is never
>> considered idle when compacting.
>>
>> Consequently the larger views are shutdown before completion. Also,
>> when a new attempt to re-compact is processed the partially complete
>> *.compact.view file is truncated. As a result these large views
>> never get fully compacted.
>>
>> If the view group compaction continued where it left off I guess
>> the impact would be minimal.
>>
>> Has anyone else seen this ?
>>
>> Regards,
>>
>> Mike
>>
>>
>>
>>
>>
>>
>
>
>
> --
> Filipe David Manana,
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."
>
Re: View group compactions shutting down in trunk
Posted by Filipe David Manana <fd...@apache.org>.
Mike,
Can you try the following patch?
http://friendpaste.com/2fT5Ne4NAQYoS9CWBsOKUW
Basically it keeps the respective database open until view compaction finishes.
Thanks for sharing this
On Tue, Sep 13, 2011 at 2:39 PM, Mike Leddy <mi...@loop.com.br> wrote:
> Hello,
>
> I've been experimenting with trunk and the new compaction daemon,
> and discovered that some of my larger views were never being fully
> compacted.
>
> Basically I am encountering a large number of these situations:
>
> [Tue, 13 Sep 2011 13:24:46 GMT] [info] [<0.173.0>] Shutting down view group server, monitored db is closing.
> [Tue, 13 Sep 2011 13:24:46 GMT] [error] [<0.155.0>] Compaction daemon - an error ocurred while compacting the view group `base` from database `XXXX-1301529600-1303948799`: shutdown
>
> I suspect that since I have a large number of other smaller databases
> (approx 2500) that are constantly being updated there is a constant
> stream of databases being closed as a result of least recently used
> recovery - I currently have max_open_dbs at 500.
>
> This does not affect database compaction as a database is never
> considered idle when compacting.
>
> Consequently the larger views are shutdown before completion. Also,
> when a new attempt to re-compact is processed the partially complete
> *.compact.view file is truncated. As a result these large views
> never get fully compacted.
>
> If the view group compaction continued where it left off I guess
> the impact would be minimal.
>
> Has anyone else seen this ?
>
> Regards,
>
> Mike
>
>
>
>
>
>
--
Filipe David Manana,
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."